Move yfinance to Yahoo API | vs | recommend yahooquery

[ValueRaider]

Where yfinance obtains data through scraping webpages (basically everything except price history), the Python module yahooquery uses the unofficial Yahoo API to fetch = faster and not encrypted.yfinance could transition to this API, but isn't it better to just recommend users move to yahooquery? GitHub, Documentation

Why reinvent the wheel?

[cmjordan42]

Assuming there's data parity with yahooquery (I've only vaguely looked at it before), I'd agree with you that there's no sense in reinventing the wheel and it's best to consolidate support efforts onto one library. Do we have anyone who uses/has used yahooquery and can comment on its quality?

[leoBitto]

in the last few days i have transitioned my app from using yfinance to use yahooquery. it is not too painful but some things change. yfinance is a bit more intuitive but yahooquery seem to be faster ( i didn't measure anything, it just seem faster to me) . some things are strange like the dataframe returned are indexed by date (see for example income_statement in yahooquery, the equivalent of financials of yfinance).
The documentation is fine, not excellent but its understandable https://yahooquery.dpguthrie.com/

the only thing missing for me is the logo_url, i really liked that...

Thank you @ValueRaider for your efforts in making the this library awesome, but i think it's time to move on sadly...

[BradKML]

If it is a speed boost but not intuitive... can someone wrap it a certain way to be more yfinance-like?

[RudyNL]

I am already converting my program from yfinance to yahooquery. Its not really difficult for an experienced programmer. I have just transferred everything around info and fast_info

yfinance is easier to understand and to handle. For unexperienced users, it is easier to start with. There is a third library yahoofinancials. I started with yahoofinancials and switched towards yfinance due to maintenance problems.

In fact yfinance, yahooquery and yahoofinancials should be merged to one project. This is a political statement. A larger community is making the project less vulnerable. Manpower and maintenance are in the long run a serious problem. The big question is not technical, but psychological. How can you create a strong community and get all users to convert to the merged community.

[galashour]

As was mentioned before, there are some 'structural' changes in the way yfinance and yahooquery return results (and attribute names of some fields are again different).

One question I wanted to bring up though:
If I understood correctly from ValueRaider, yahooquery uses Yahoo's undocumented API.
What would happen when Yahoo will start encrypting this data too?
Also, I thought ValueRaider mentioned that yahooquery uses some 'hidden tables' and again, while Yahoo kept these for a long time, this could change.

So - while there is certainly merit in having 1 solution with bigger community, when something will break we will have no alternative till a fix is available, while currently we do have another solution (that may be less affected). As example, if the hidden tables will be removed, yahooquery users will be able to use yfinance as alternative (and vice versa).

[RudyNL]

How vulnerable are we for changes by Yahoo. In case of web scraping we are very vulnerable. Yahoo can at any moment decide to change their structure and layout. I guess that the Yahoo website is in some way build on the Yahoo API. The API is probably more stable.

I think a two layered structure is the best. There is a user interface layer like yfinance and yahoofinancials and a data retrieval layer like yahooqueries or the web scraping part of yfinance. For end users you want to keep the user interface as stable as possible. The data retrieval layer should be as flexible as possible. Any data provider Yahoo, Google or who ever could be used as data source. For the end user its irrelevant where the data comes from. More data sources are making the software less vulnerable.

I am not so afraid of Yahoo blocking the usage of the API. The number of data sources is increasing. Brokers are also offering APIs to their clients. Yahoo is at this moment the Google of the financial world. If they are blocking too much data, they will lose this position. There is already a Google Finance https://www.google.com/finance/

[dpguthrie]

Hey all - thanks for tagging me in this thread!

Let me just start by saying that very much like @JECSand (and thank you for the kind words!), I wrote yahooquery awhile ago when I had a lot more time (and much less python capabilities). There are certainly things I would've done differently with the knowledge I have now.

As for the current state, I was unaware that YF had started encrypting the javascript on the page to (probably) discourage web scraping. I know in the very recent past they also were blocking any requests from python without specifying a valid user-agent. Also, it seems they're also able to spam the historical pricing API with erroneous data (this is purely anecdotal). All that to say, I don’t have a lot of trust that YF won’t continue to make it harder for us to get data from them either through scraping or an API.

I don’t have any strong opinions on what’s been said above, but I am more than happy to work with others to find a suitable solution, whatever that may be.

[dpguthrie]

@galashour I haven't checked for parity between the two, but I would be surprised to find that it wasn't at least close. The two that you mentioned: longName can be found in the price property and the quote_type property and mostRecentQuarter can be found in the key_stats property. Those properties map to what YF calls modules in their API.

You can use the search bar in the docs or do a ctrl+f here

[ValueRaider]

@galashour @dpguthrie If you don't mind, I'll edit and delete the chatter about longName etc, to keep main discussion focussed.

[ValueRaider]

Maybe we move yfinance over to the official API then give a 6 month deprecation warning to get users moved to yahooquery or a new derivative project? I can do the same on YahooFinancials. There's 17k projects dependent on yfinance for example, giving them time and forewarning to move would likely be very welcome. Additionally, perhaps we can all work on yahooquery / derivative project to make it handle more like yfinance.

@JECSand Pulling this part of your comment out, I think should be a separate thread - how to handle immediate migration to API.

Ideally we just tell users to switch to yahooquery, but I know it's not a perfect drop-in replacement. E.g. it returns financials tables transposed. Might be other divergences. I see two possibilities:

• work with yahooquery to reduce the divergence then recommend simple switching
• yfinance uses yahooquery as a backend, then transforms the data output to meet user expectations e.g. transpose financials. Avoids breaking all those dependent projects and data pipelines

[dpguthrie]

Hmm...not sure I would want to just wholesale change everything on the yahooquery side to account for the differences. We could certainly introduce some "yfinance wrappers" over what's existing. Also, the structure of the dataframes returned in yahooquery map more to my mental model of how that data would be stored in a database than yfinance (e.g. financial line items as columns vs. dates).

[RudyNL]

The backend is in fact the data source. There could be more competing backends. The seperation of frontend and backend makes the complete system more flexible. If for any reason Yahoo stops delivering data, the backend can be replaced by another backend to retrieve the data from elsewhere.
@ValueRaider proposal to transform yfinance to a frontend/user interface is a very sensible solution.
We are fixated on Yahoo. Are there other datasources which can be used?

[ValueRaider]

I did not propose to allow yfinance to fetch data beyond Yahoo. My plan is it only fetches data from Yahoo Finance, directly or indirectly. A generic solution should be a fresh new project, with a better ownership structure to avoid 'missing owner' problems i.e. GitHub organization.

Backend is relative. Backend of yfinance currently is webpage scraping (exceptions: price history and Ticker.get_shares_full). This likely to break soon. It's possible to ease user migration to yahooquery, as I listed above.

[ValueRaider]

@dpguthrie Regarding "yfinance wrappers" - do you have a clear idea of how you want this implemented? If you provide pseudocode / prototype example then we can implement most of it. If not, we (open invite) should brainstorm - my suggestion is a global toggle variable e.g. yq.yfinance_compatibility = True then the internals handle formatting.

[dpguthrie]

Open to suggestions here, of course, but was thinking along those same lines as well. Here's a super simple example with balance sheet (I know there are other differences besides the structure of the dataframe that you may want to address as well):

def balance_sheet(self, frequency: str = 'a', trailing: bool = True):
    df = self._financials('balance_sheet', frequency, trailing)
    if self.is_yfinance:  # property set when initializing Ticker class, default is False
        return df.T

    return df

[ranaroussi]

@ValueRaider Currently, yfinance has the largest userbase by far with 600K+ downloads/mo, according to PyPi. So I would be very careful to introduce any breaking changes or change how people interact with it (I'm still getting angry emails about the pdr.override removal).

Re:

"This likely to break soon. It's possible to ease user migration to yahooquery, as I listed above."

Given that yahooquery is a wrapper for the unofficial yahoo api* - it also carries the risk of breaking.

What I think we should do is start a gradual move towards using Yahoo's API (official and unofficial) as opposed to introducing new dependencies (especially ones like yahooquery that rely on selenium for some things).

Another thing that's been on my imagined "roadmap" is to integrate a storage mechanism (maybe with pystore?) to be optionally initialised using yf.enable_store('path/to/store'), and have all queries store data there to be used in future queries - enabling "offline" support that expands over time.

Re:

"'missing owner' problems"

I've created an organization named financepy. How about we start by creating a project with some sort of a roadmap and plan this properly?

[akraievoy]

IMO: if yahooquery brings in too many dependencies, then only the relevant API consuming code to serve as a drop-in fail-over for the UI scraping should be ported over to yfinance, the results generated should be quietly mapped to existing yfinance APIs -- this way we get both improved data access and minimal end-user astonishment

as for caching: I happen to have some of it implemented locally but this code is not too reusable overall

another interesting gotcha with these caches [is] expiration policy -- most things should not expire [with ordinary within-week timeout] during the weekend

[ValueRaider]

@akraievoy Yup caching is tricky, best to incorporate exchange schedule, and you didn't even mention backporting new dividend adjustments (harder than you'd think). Check out yfinance_cache, mostly a price data cache so I might rename.

[ValueRaider]

What I think we should do is start a gradual move towards using Yahoo's API (official and unofficial) as opposed to introducing new dependencies @ranaroussi

What about the third option - recommend users switch to yahooquery? After we have contributed changes to yahooquery so it can be a seamless drop-in replacement (currently is not), probably less work than porting yfinance to the API.

[RudyNL]

I switched from yfinance to yahooquery. Its not very difficult and I personally prefer to be closer to the datasource (API). yahooquery is much slower due to the missing caching. I think that without caching yahooquery will beat yfinance. To get a working program is easy, to get it fail-safe is much harder. It took me a week to get my program at the same service level as with yfinance. The difficulties are for example with the missing data conventions like None for yfinanceand an error string for yahooquery.

My opinion: for a good programmer I would advise to switch to yahooquery. The problem is the large userbase of less experienced users of yfinance.

[BradKML]

If so then what is stopping yahooquery from adopting cache?

[asafravid]

My 2 cents are that yahoo query has the data only in different formats
Hence I propose a conversion mechanism here
Which works well here (https://github.com/asafravid/sss).

it allows the scanner to use the yfinance data formats in an already existing functionality which relies upon those.
#1454

A sort of Adapter, in some sense