Let's fix the financials data!

[ValueRaider]

So the financials data has been broken for a long time, I'm sure everyone knows by now. But a fix was proposed a year ago so this should be quick to sort out.

The blocker now is testing. The fix is significant so I'm reluctant to merge without thorough testing. Can others download and test, give feedback? The fix is in branch fix/financials_data. This other discussion explains how to "install" it.

The fix contains 2 big changes to the annual data tables:

1. Fixing the table contents (obviously) - is anything still wrong?
2. Make table index hierarchical, resembling what Yahoo displays - is this actually helpful? Or annoying?

Will port fix to quarterly tables once everyone OK with fix.

Interface changes:

• Ticker.financials renamed Ticker.income_stmt
• Old bad data available via _legacy suffix, e.g. Ticker.income_stmt_legacy

[ValueRaider]

Tagging anyone who's raised a related issue:
@cesar-moya @HoneyJamFinance @hkristof03 @NicsunXnus @Rapid1898-code @prischon @DavidIsrayelyan

[ValueRaider]

@xwu64 @longsondt @Slepetys

[ValueRaider]

@focus8 @byagiz @jupper

[ValueRaider]

@walterlsb @teeejer @git-shogg

[ValueRaider]

@asafravid

[git-shogg]

If it would make it easier I am happy to restore the original attribute names? I can do this next week?

[ValueRaider]

You can ignore that old comment from me. Keep the renaming. Codes are going to break anyway from the new rows.

[git-shogg]

Ok no worries 👌

[git-shogg]

@ValueRaider if this goes ahead can you please ensure I am included as a contributor to this fix?

[ValueRaider]

Of course!

Do you have bandwidth for significant development? E.g. refactor (the 3 code blocks look similar), and investigate fixing the quarterlies?

[ValueRaider]

@git-shogg I've spotted a bug with ticker IBN - financials on website but code not generating tables. I'll see if I can get you access to this project so easier to fix.

[git-shogg]

Hi @ValueRaider, I can't easily fix this right now but I can see the bug. Jump into utils.py and then go to the format_annual_financial_statement (should be line 230), replace the if statement in here with:

if (ttm_dicts != None and ttm_dicts != []) and (ttm_order != None and ttm_order != []):

[aliencaocao]

I have a question regarding

yfinance/yfinance/utils.py

Line 288 in 22b87f9

start_dt = _datetime.datetime(2016, 12, 31)

Why the start_dt is fixed to _datetime.datetime(2016, 12, 31)? Won't this cause any data earlier than that date to not be returned? A safer value is probably 0 since yahoo always returns the most recent 5 quarters anyways.

[ValueRaider]

The earliest data possible to return right now is maybe 2018. All data will be returned.

[aliencaocao]

Oh, did not realise that. Thanks for the clarification.
To new contributors who do not realise this though - it can be confusing. I still suggest setting it to 0 for sake of clarity.

[ValueRaider]

Well it's 4 years or 5 quarters, I figure any contributor knows this.

[asafravid]

@ValueRaider right on spot

[ValueRaider]

@git-shogg Is it truly necessary to reproduce the nested structure in the table index? Why can't user just refer to Yahoo.com for relationships?

[git-shogg]

Yes, a user could just refer to Yahoo.com for the relationships. However, setting the script to dynamically determine these relationships for absolutely any financial statements is more beneficial to end users. The reasons why are:

1. This will work for any financial statement (line items could differ from company to company - this script handles this and provides users with what they need to understand how it all fits together).
2. Users can utilize this package to understand the financial statement construct (not just the line item values which are less useful with less context - e.g. being able to automatically work out that in the case of BHP Group Limited Salaries and Wages makes up a part of G&A expenses which are a subset of SG&A which is a subset of Operating Expenses).

[ValueRaider]

@aliencaocao FYI

@git-shogg Can you share actual code how you currently use structure? I wonder if a proper tree structure is better, with search & children functions.

[ValueRaider]

I have merged @git-shogg improvements but with the MultiIndex disabled, it needs more thought #1128. Will ship in a release soon.