With Generative AI, Data Analysis Will Never Be the Same

But humans will still be helpful

My first paying data analysis job was more than 50 years ago. And for the first 45 years or so of my career in this field, it was a pretty consistent set of tasks. You put your data into rows and columns, figured out what to do with missing data, specified what type of analysis you wanted on what variables, and fed it all into a statistical analysis program. You hoped you didn’t make any spelling missteaks (sic), because you’d have to do it all over again. Interpretation of the results, of course, was up to the analyst, as was the explanation of the implications for action for your decision-maker boss.

How GenAI Can Change Data Analysis

But generative AI is clearly changing the process of data analysis. I’ve been experimenting with quantitative data analysis using ChatGPT for several years now. I was certainly impressed by the LLM’s ability to generate Python code to analyze structured data and create machine learning models. With only a short, two-line prompt it could do the following:

• Figure out (in a CSV file) which were the independent variables to analyze as predictors of the dependent variable; all it had to be told was the label of the dependent variable.

• Determine how best to handle missing data

• Do a bit of feature engineering to see if it could create a better fit for the model

• Try several different algorithm types and run a little competition among them;

• Describe which features had the greatest importance in the best-fitting model.

Even more impressively, the LLM could provide a page or two of recommendations for a manager seeking to maximize or minimize the dependent variable (I often used an attrition dataset from the telecom industry). It might suggest, for example, what products a company should offer in what bundles in order to reduce attrition.

There was only one problem with this approach to data analysis. I noticed that minor differences in prompt language led to differences in the results. I’m not experienced enough in Python to determine why this happened, but it made me hesitate to recommend ChatGPT-based data analysis to others. Perhaps that’s why LLM-driven machine learning isn’t practiced all that much.

Enter Claude

A few months ago I tried uploading a CSV dataset to Claude for analysis, but it told me that it couldn’t accept uploaded CSV files. But over the December holidays I had some data to analyze for a survey on the economics of AI. My collaborator on the project, Laks Srinivasan, suggested that we see what Claude could do with the analysis. He said that Claude still had issues with uploading CSV files, but that it seemed to accept SPSS data files, which we had received from the survey provider. Laks fed Claude the SPSS files and gave it some prompts with context about our study and what we were expecting to address in the reports and articles emerging from it.

And in general, Claude did an amazing job. It quickly identified what it thought were the most interesting and important findings. It created an outline for the final report and would no doubt have written a draft of the report had we let it. It performed crosstabulations of several variables, and identified some subtle findings that I am not sure I would have noticed.

We all, including Claude, were heavily focused on one issue. Laks and I had previously identified the survey questions and variables involved in AI economic maturity. Claude made that the focus of its analysis. As Claude pointed out, “Every consulting firm has an AI maturity model. Most are useless….We took a different approach: economic value creation maturity.” (bold font supplied by Claude).

Claude also found it very important to point out that many companies in the survey were “stuck” in the middle of the economic maturity model, particularly on Stage 3 (in which economic value from AI is measured but not aggregated across the company).. It suggested several times that the surveyed companies had been stuck there for a median of six years. It commented:

“Nearly one-third of organizations (30.1%) are stuck at Stage 3 for a median of 6 years. They achieve respectable value (44.2%) but can’t aggregate because each use case is measured differently. The escape requires standardization, not more measurement.”

I found that conclusion interesting, since we didn’t ask companies in the survey how long they had been at each stage. When Laks asked Claude how he came to that conclusion that many companies were “stuck at Stage 3,” it replied that 6 years was the median time Stage 3 companies had been using AI. Claude did not acknowledge that companies could have been at a higher stage and fallen back, or that they had perhaps been in Stage 3 for a short time.

Claude may have been wrong about this finding, but it was never in doubt and continued to push that idea. To its credit, however, when we asked for an explanation it said that these companies “have achieved a fair amount of value but can’t standardize value across use cases.” The evidence it cited included:

• 52.8% cite “lack of AI-ready data”

• 50.5% cite “technology difficult”

• 49.2% cite “lack of framework”

• 38.3% cite “leaders lack understanding”

That logic was moderately compelling, so we left the “stuck at 6 years” in the report.

It also turns out that Claude is also a big evangelist for agentic AI. Perhaps that’s not surprising, as it represents an important direction for the LLM’s future. Even though a relatively small fraction (about a third) of companies in our survey were using agentic AI, and those using that technology got the least amount of high value from it (2%) than any other technology, Claude continued to be a strong advocate for it as a “sophistication signal.” Claude commented:

While the first three AI types are table stakes (80%+ adoption), Agentic AI is the sophistication signal. Organizations using Agentic AI consistently outperform across multiple value dimensions—not because Agentic AI itself creates all the value, but because deploying Agentic AI requires organizational maturity (measurement frameworks, governance, risk management) that also drives value in other AI initiatives.

We weren’t sure that any of these conclusions were true, and they were certainly overstated. The high value of AI that agentic adopters are getting is almost certainly coming from other AI technologies. But its comments were interesting, and we did hint at the “sophistication signal” in the report.

A Smart but Pushy Intern

Others have drawn this analogy before, but on our project Claude was akin to a very smart and pushy intern who saved us considerable time. We were able to engage in a dialog with it—in many cases an effective one. The LLM sometimes went too far beyond the data, and rarely backed away from the ideas it advocated. Once we did point out that it was over-generalizing, and it apologized for this behavior. But despite a few glitches, our work was definitely better as a result of having Claude on the team.

Will Claude replace a lot of data scientists or data analysts? I think the most likely effect of these capabilities will be the democratization of data analysis and some data science. More people will be analyzing data, not less. Some only moderately qualified practitioners may need to move into other fields. However, highly skilled analysts and data scientists will still be necessary. It’s occasionally important for somebody to tell Claude and its LLM relatives what to do, to figure out whether it’s making sense, and to push back on this pushy intern when it ventures too far away from the data.