Innovation Intern

Supernova Technologies is a fintech startup doing securities based lending. By the summer after my freshman year most of their core services were already built, and AI in business was just starting to become a real thing, so I got to see the early days of it inside a company and be the first person to work on those problems there.

The first problem was questionnaires. A fintech holding a lot of financial data receives due diligence and compliance questionnaires from the institutions it works with, and the IT side of the company had to answer every one by hand. Each institution asked a slightly different set of questions, or the same questions worded differently, and a single questionnaire could take a couple of days to finish. Before large language models there was no real way to automate it.

The second problem was document parsing. Customers submit W2s, insurance documents, proof of income and similar forms, and each has to be turned into the company's own structured format, usually JSON, before it can be used downstream. The company was using Amazon Textract for this, and at the time it was both expensive and not nearly as accurate as such services are today. Textract is built for every document type, while the company's documents were a small set of highly specialized forms, so the question was whether something purpose built for those forms could do better for less.

The question answering service end to end. IT sent me their reference documents as PDFs, I converted them into a vector database, and for each question the service retrieved the most relevant passages, passed them with the question to a large language model, and returned a yes or no answer with the reasoning and the source text behind it. The model could not take full ownership of a questionnaire, but it turned answering from a finding problem into a verification problem: the reviewer checks a cited answer instead of hunting for it and writing the justification.

The document parser. The first approach was to extract all of the text from a PDF and prompt a language model to pull out the required fields and return JSON. It was cheaper than Textract but not accurate enough, even across different providers and prompting strategies. The approach that worked combined the specialized format of the documents with some projection geometry. Once a W2 is resized and standardized, the name is always in the same corner and the address is always in the same place, so each document type gets a template of bounding boxes defined relative to the page size. Extraction then runs inside each box, where it is looking for one specific thing, and the result is assembled into JSON, with or without a language model pass afterward.

This was my first time in industry and I did not know how to approach problems this size, so the first step was asking. I brought what I was thinking to the engineers around me, asked whether it was worth pursuing, and used their answers to narrow down which questions to answer and which directions to explore.

The second part was working within constraints. One suggestion was to fine tune a purpose built model on the company's own data, which in the long run might have been the most impactful option, but for a startup it was hard to justify the cost, and fine tuning done badly can make a model worse. Those constraints made the problem more enjoyable. The question became what we already knew about the documents, what assumptions were safe, and which invariances in the document types we could exploit instead of paying to learn them.

Two services that were easy to understand, cheap to run, built on open tooling, and accurate. The question answering service reached about 97 percent accuracy before human review, and the bounding box parser reached about 90 percent on complex financial layouts, up from about 40 percent with the text only approach. The benchmark that produced those numbers is what let new strategies and model versions be compared on the company's own documents rather than on a vendor's.

For me, this is the experience that pulled me into machine learning and deep learning, and it is where my interest in reinforcement learning started. It also added OpenCV and a good deal more Python to what I could work with.

For question answering, the weakness of the early models was hallucination, so I would add a measure of doubt: a confidence score based on the entropy of the model's response, compared against a baseline so that uncertain answers are flagged for the reviewer. I would also add automatic checking of citations, so that a cited passage is verified to support the answer rather than trusted.

For document parsing, I would spend more time researching models and providers built for that specific problem. Looking back, the tools that exist now for this are remarkable, and they were not at the time, but more research up front would still have been the right call. The same goes for the summer as a whole. I was inexperienced, and my pattern was to describe an idea to my team lead, hear yes, and then not talk to him again until it was done, even when I was struggling. I would break the projects down properly, walk through the design and the alternatives with him, and check in while the work was in progress.

The biggest lesson is not to rush into code. On a research problem, spending a week thinking and reading before writing anything is progress, not delay. Use everything you already know, look hard for the invariances in the problem, and find out whether someone has already solved it before you start.