Generating package summaries with GPT #2368
Replies: 5 comments 10 replies
-
ResultsAll results are an unprocessed README file copied and pasted from the rendered Markdown with the prompt above. No editing has been done to the results. Alamofire
SemanticVersion
Satin
FlyingFox
ArgumentParser
Swift Crypto
|
Beta Was this translation helpful? Give feedback.
-
|
If you go forward with this, then it must be opt-in, not opt-out, for package maintainers. I personally would be very upset if I learned that descriptions that may or may not contain confidently wrong statements had been automatically generated for my packages without asking me first. |
Beta Was this translation helpful? Give feedback.
-
|
I can’t say whether it will be opt-in or opt-out right now, but we take your point seriously. We actually just talked a little about opt-in/opt-out in our next podcast episode that we recorded today and which will be released on Thursday if you’d like to listen. I believe the chances of “confidently wrong” answers are much lower than with other GPT responses because we’re not asking the API to generate original writing. We’re passing in README content and asking it to refine it. I’m not saying that it couldn’t happen that it could get things wrong, but from my experimentation so far the chances are low. Of course, we will look at significant numbers of results carefully before making any decisions on if/when/how we roll a feature like this out. — I just ran your README files through the API and here are the results:
It doesn’t do very well with logic parser or Existential Graphs Foundation but it’s working off quite a small amount of text in the original readme files. I’m sure it would give better results with a little more input data. Finally, as mentioned above, we’ll also provide an override in the |
Beta Was this translation helpful? Give feedback.
-
Thinking about this a little more, there's a good way to avoid any issues with it trying to generate the "40 words" that the prompt asks for if there are less than 40 words in the source text. We just don't run it on short README files and the full text of the README becomes the summary. |
Beta Was this translation helpful? Give feedback.
-
|
Naturally running READMEs which are entirely unstructured and unstandardised through an LLM is going to result in some unexpected cases. I'm not sure if we have analytics for this, but I've definitely seen packages with no README, packages with READMEs in different languages (such as Chinese), boilerplate READMEs (either with no information in, or just the standard "This is a Swift Package" template). There are READMEs with FAQs in, TODO lists, version history and testimonials, links to podcast episodes and books. There is so much variety here and areas where GPT could get confused. But no system is perfect, and I think I've seen enough examples here (and separately) where it's output very strong descriptions even in some of these tough scenarios. I think the prompt could be tweaked over time to wean out bad results. But I think there's also stuff we can do with the implementation to make it safer including filtering out bad READMEs, and potentially starting with only the top 500 packages (based on internal scoring) to test it out. I also think introducing a button to the UI to report a package summary could be handy, one which simply opens a GitHub issue with a pre-filled form. This would help surface these bad results so we, as a community, can re-assess and make changes to the prompt, to the README, and if necessary override using the SPI YML. |
Beta Was this translation helpful? Give feedback.
-
|
To be clear, on the whole I am very supportive of this and think it will provide a tonne of value to the community to make search results better and make it easier to find the right package. |
Beta Was this translation helpful? Give feedback.
-
|
These are great points, thanks James! I just ran a your examples through to see how it did and these were the results:
The best package ever… LeftPad:
We also chatted about opentelemetry-swift after you posted this, which contains only a very short package description followed by lots of potentially distracting data:
I’m constantly surprised by how good they are. |
Beta Was this translation helpful? Give feedback.
-
|
Heard about this conversation through the SPI podcast - wanted to chime in. I think using it to create a 2-3 sentence summary for a package might be a fantastic use case, specifically to supplement search results. That said, I think the summary result should be something that a package developer can override - perhaps a couple sentence summary in the structure of the SPI.yml file? I also think the field should be clearly identified - maybe with a background color, or a caption-like subtitle underneath it (my design skills suck, so I'll defer to nearly anyone else about good affordances here). The idea being that when generated by an LLM, the summary should be highlighted as dynamic content. And somewhat related, using LLM generation to deal with stemming issues in the search is absolutely brilliant. A longer summary - as a generated and stored text field within the index, and then in turn used with the Postgres style search mechanism (the classic stemmer/ranking mechanisms) - would, I suspect, improve ranked relevance for those quirky results where we've got two or more words CamelCased together that are easy to visually disambiguate, but which that stemmer fails with - and extending that system in Postgres is an unfortunate challenge. I don't know the details about ChatGPT as a tool - but I'm wondering if perhaps it's also multi-lingual (Whisper, also by OpenAI clearly is). If so, then that could also potentially up the value of summarized search results if we can ask it translate into English for the search-only-summary view. I wouldn't want to display that translated/search-focused result - but I suspect it may solve the issue of getting a reasonable summary from some of the packages that have a README in another language (the ones I remember being Mandarin - but it wouldn't surprise me to see Japanese or various European languages either). There's some downsides with that - but I think overall the results from search would be notably improved. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the feedback, Joe!
Absolutely. I mentioned this in the original post. The description field in the SPI manifest would be essential before we do this.
This is a very good point. Certainly GPT is "fluent" (whatever that means in this context! 😂) in multiple languages and I just had it answer a question in Mandarin and it did well (I only checked the result with Google Translate, it would need better testing than I can do). I think Mandarin is the only non-english language I've seen in README files, can you think of any other examples? |
Beta Was this translation helpful? Give feedback.
-
|
I can't quite believe this, but I just ran it through for LJTool and Srt2BilibiliKit with no modifications to the prompt and it did an outstanding job. Yes, it translates the summary into English, but a modification to the prompt would fix that. For LJTool:
I've not used LJTool, but translating the first paragraphs of the README with Gooigle Translate, that summary seems good. The Google Translate text was this:
For Srt2BilibiliKit:
The word danmaku seems odd in this context, but from the Google Translate below, I guess it's coming from "barrage". Running the relevant bit of the README through Google Translate gave this:
|
Beta Was this translation helpful? Give feedback.
-
|
I just had a quick try to get the prompt to summarise without changing the language and this worked well:
Added words are in bold. The output of the same input text for LJTool was then:
Which, passed through Google Translate, is:
That is remarkable 🤯 |
Beta Was this translation helpful? Give feedback.
-
|
Mandarin was the only other language I saw in READMEs (while doing some of that earlier search work), although honestly I expected to find some Japanese if I kept looking, but the packages I've found from folks in Japan appear to write their READMEs in English. I've no idea how much the quality of training and weights for the underlying LLM makes a difference in practical outputs, but one of the ideas for cost reduction that I had yesterday was to potentially use one of the "open source clones" - there's quite a number following Meta's Llama weights disclosure, although not as neatly packaged with an explicit API. But as a potential upside, they can run in a single computer and not require external hosting, with the main constraint I've seen being some published models being licensed for non-commercial use only. |
Beta Was this translation helpful? Give feedback.
-
|
Finally got access to a GPT-4 API key and did some more testing with this tonight. The results are different, but still very good. There's much more structure to the prompt, which is great. All of these results were generated from raw, unprocessed markdown files so we would not need to render or transform anything. This is with a smaller word limit than above, but it really struggles with word limits 😂 PromptYou are a technical editor that summarises Markdown input. Output
|
Beta Was this translation helpful? Give feedback.
-
|
Costs for this may be more reasonable than we thought, too. I submitted about 50 API requests with the Alamofire README file (which is quite large) and then a few more with the examples above and it cost less than $2. Extrapolating that for 5,500 packages would mean we'd be up against a ~$200 cost for initial ingestion and probably $15-$20/month ongoing. |
Beta Was this translation helpful? Give feedback.
-
|
Very exciting. I'd love to see this added, think it'd be huge for search/discoverability. |
Beta Was this translation helpful? Give feedback.
-
We are not trying to find a use for GPT in SPI, but there are a couple of uses (one in this thread and one in #2369) where it could be useful.
The first and most practical one is in README summarisation. GPT is good at summarising text, and we don’t have good, concise descriptions for most packages in the index. We have package name and sometimes a one sentence description, and then the full README file which is too long and often contains much more than a package description.
Having reliably good summaries would be very useful for package search results, category and author lists, and also in a slightly redesigned package page metadata area. It would also give us another relevant field for search results without bringing in the entire README file.
It’s important to note that if we do include package summaries as metadata, they would be overridable by humans via a new the
.spi.ymlfile. We would always prioritise human entered text over AI-generated.We’re not quite at “New Issue“ with it yet, but we’re convinced enough that it’s worth a discussion.
Costs
Running v3 queries is very cheap and with some back of the envelope calculations we could summarise every README in the current index for under $50. We then would only need to change the summary when we see README changes, and at most something like once a week.
GPT4 queries are much more expensive, so the results would need to be much better. TBD when we get the new API key.
Results
These results are were made using the
text-davinci-003model. We are on the waiting list for a GPT-4 API key. The full text of a README was entered without any pre-processing and processed with the following prompt:Results follow this post.
Alternatives
We did some experimentation with the Kagi summariser but it produces summaries that are too long and always include installation instructions. A problem that GPT had too, but could be worked around with the prompt.
Beta Was this translation helpful? Give feedback.
All reactions