Impact of a Deployed LLM Survey Creation Tool through the IS Success Model

arXiv 2025 44 method

TLDR

Deployed LLM survey creation tool evaluated via IS Success Model, with hybrid evaluation and safeguards.

Reasoning

The paper's strength lies in its real-world deployment and practical evaluation using the IS Success Model, offering a hybrid evaluation framework and safeguards. However, its scope is limited to survey creation, not general automated scientific discovery or autonomous research agents.

Read-first score

Read-first score 44, weighted from topical fit, citation, graph, method, reproducibility, and recency signals. Original total remains 34.

Recency 8%

86.7

Uses a gentle age decay so recent papers surface without erasing older foundations. 2025

Methodology quality 25%

Screens visible abstract and analysis fields for experiment, dataset, baseline, metric, and limitation evidence. markers=evaluation,experiment,result

Reproducibility 25%

Screens links and visible text for paper, code, dataset, artifact, and repository signals. pdf=True; code=False; dataset=False; markers=none

Topical relevance 42%

28.3

Uses existing LLM keyword relevance scores normalized to 0-100. AI scientist,automated scientific discovery,autonomous research agent,automated research,literature review agent,survey generation,automated experimentation,experiment design agent,AI for scientific research,paper writing agent,research automation,scientific discovery agent

Field roles

FrontierMethodology anchor

Rank sensitivity

Stability: volatile; rank range: 15.

Keyword Scores

survey generation

AI for scientific research

automated research

research automation

AI scientist

automated scientific discovery

autonomous research agent

literature review agent

automated experimentation

experiment design agent

paper writing agent

scientific discovery agent

Deep Analysis

Innovations

First application of the IS Success Model to a generative AI system for survey creation
Proposal of a hybrid evaluation framework combining automated and human assessments
Implementation of safeguards to mitigate post-deployment risks and support responsible integration

Methodology

The study deploys an LLM-powered survey creation tool in a real-world setting and evaluates it using the DeLone and McLean IS Success Model, employing a hybrid framework of automated and human assessments.

Key Results

The abstract does not report specific experimental results; it highlights contributions and evaluation approach.

Resources

Paper