Merge pull request #53 from noirbizarre/epci

Initial intercommunalities support

CHANGELOG.md

@@ -1,5 +1,9 @@
 # CHANGELOG
 
+## 10.0.0.0 - 2017-08-18
+
+* Integrate intercommunalities (ie. french EPCIs) historical data
+
 ## 9.0.1 — 2017-06-20
 
 * Fix 2A, 2B and 17 departements’ parents references.

README.md

@@ -7,7 +7,7 @@ It might be useful if you have to deal with redirections and is in use by the [g
 
 ## Usage
 
-If you’re only interested in generated data, check out the `exports` folder which contains CSV files related to [regions](exports/regions/), [counties](exports/departements/), [overseas collectivities](exports/collectivites/) and [towns](exports/communes/). There is a dedicated documentation at these places.
+If you’re only interested in generated data, check out the `exports` folder which contains CSV files related to [intercommunalities](exports/epci), [regions](exports/regions/), [counties](exports/departements/), [overseas collectivities](exports/collectivites/) and [towns](exports/communes/). There is a dedicated documentation at these places.
 
 
 ## Sources
@@ -16,6 +16,8 @@ Source files are coming from the [INSEE downloads page](https://www.insee.fr/fr/
 
 Additionaly, files containing the population for almost all towns has been computed too in the `sources` folder. They are coming from [a XLS dataset](http://www.insee.fr/fr/ppp/bases-de-donnees/recensement/populations-legales/pages2015/zip/HIST_POP_COM_RP13.zip) provided by [INSEE](http://www.insee.fr/fr/ppp/bases-de-donnees/recensement/populations-legales/), manually completed with Wikipedia data for [Lyon](https://fr.wikipedia.org/wiki/Arrondissements_de_Lyon) and [Marseille](https://fr.wikipedia.org/wiki/Secteurs_et_arrondissements_de_Marseille) districts, and converted into CSV.
 
+[Intercommunalities source files are fully documented into their folder](sources/epci).
+
 
 ## Development
 
@@ -31,6 +33,14 @@ Optionally, you can specify a date to only export towns valid at that given date
 
 It will be generated within the `exports/communes/` folder with an explicit name.
 
+To also generate the intercommunalities, you need to add the `--intercommunalities` flag.
+
+ $ python -m geohisto --intercommunalities
+
+The whole process takes about one hour and a half to generate both towns and intercommunalities exports (on a core i7 with 16Gb RAM).
+You may add some extra output to see the progress by setting the verbosity to `debug`:
+
+ $ python -m geohisto --intercommunalities -v debug
 
 ## Tests
 

exports/communes/communes_head.csv

@@ -3,6 +3,7 @@ fr:commune:01001@1942-01-01,01001,1942-01-01 00:00:00,9999-12-31 23:59:59,L'Aber
 fr:commune:01002@1942-01-01,01002,1942-01-01 00:00:00,9999-12-31 23:59:59,L'Abergement-de-Varey,,,fr:departement:01@1860-07-01,236,0
 fr:commune:01003@1942-01-01,01003,1942-01-01 00:00:00,1973-12-31 23:59:59,Amareins,fr:commune:01165@1974-01-01,,fr:departement:01@1860-07-01,NULL,330
 fr:commune:01004@1942-01-01,01004,1942-01-01 00:00:00,1955-03-30 23:59:59,Ambérieu,fr:commune:01004@1955-03-31,,fr:departement:01@1860-07-01,NULL,100
+fr:commune:01004@1955-03-31,01004,1955-03-31 00:00:00,9999-12-31 23:59:59,Ambérieu-en-Bugey,,fr:commune:01004@1942-01-01,fr:departement:01@1860-07-01,14359,0
 fr:commune:01005@1942-01-01,01005,1942-01-01 00:00:00,9999-12-31 23:59:59,Ambérieux-en-Dombes,,,fr:departement:01@1860-07-01,1635,0
 fr:commune:01006@1942-01-01,01006,1942-01-01 00:00:00,9999-12-31 23:59:59,Ambléon,,,fr:departement:01@1860-07-01,108,0
 fr:commune:01007@1942-01-01,01007,1942-01-01 00:00:00,9999-12-31 23:59:59,Ambronay,,,fr:departement:01@1860-07-01,2503,0
@@ -14,12 +15,14 @@ fr:commune:01012@1942-01-01,01012,1942-01-01 00:00:00,9999-12-31 23:59:59,Aranc,
 fr:commune:01013@1942-01-01,01013,1942-01-01 00:00:00,9999-12-31 23:59:59,Arandas,,,fr:departement:01@1860-07-01,158,0
 fr:commune:01014@1942-01-01,01014,1942-01-01 00:00:00,9999-12-31 23:59:59,Arbent,,,fr:departement:01@1860-07-01,3410,0
 fr:commune:01015@1942-01-01,01015,1942-01-01 00:00:00,2015-12-31 23:59:59,Arbignieu,fr:commune:01015@2016-01-01,,fr:departement:01@1860-07-01,495,331
+fr:commune:01015@2016-01-01,01015,2016-01-01 00:00:00,9999-12-31 23:59:59,Arboys en Bugey,,fr:commune:01015@1942-01-01;fr:commune:01340@1942-01-01,fr:departement:01@1860-07-01,631,341
 fr:commune:01016@1942-01-01,01016,1942-01-01 00:00:00,9999-12-31 23:59:59,Arbigny,,,fr:departement:01@1860-07-01,427,0
 fr:commune:01017@1942-01-01,01017,1942-01-01 00:00:00,9999-12-31 23:59:59,Argis,,,fr:departement:01@1860-07-01,422,0
 fr:commune:01018@1942-01-01,01018,1942-01-01 00:00:00,1970-12-31 23:59:59,Arlod,fr:commune:01033@1956-10-19,,fr:departement:01@1860-07-01,NULL,310
 fr:commune:01019@1942-01-01,01019,1942-01-01 00:00:00,9999-12-31 23:59:59,Armix,,,fr:departement:01@1860-07-01,21,0
 fr:commune:01020@1942-01-01,01020,1942-01-01 00:00:00,1964-06-03 23:59:59,Arnans,fr:commune:01125@1942-01-01,,fr:departement:01@1860-07-01,NULL,310
 fr:commune:01021@1942-01-01,01021,1942-01-01 00:00:00,1956-10-18 23:59:59,Ars,fr:commune:01021@1956-10-19,,fr:departement:01@1860-07-01,NULL,100
+fr:commune:01021@1956-10-19,01021,1956-10-19 00:00:00,9999-12-31 23:59:59,Ars-sur-Formans,,fr:commune:01021@1942-01-01,fr:departement:01@1860-07-01,1387,0
 fr:commune:01022@1942-01-01,01022,1942-01-01 00:00:00,9999-12-31 23:59:59,Artemare,,,fr:departement:01@1860-07-01,1175,0
 fr:commune:01023@1942-01-01,01023,1942-01-01 00:00:00,9999-12-31 23:59:59,Asnières-sur-Saône,,,fr:departement:01@1860-07-01,69,0
 fr:commune:01024@1942-01-01,01024,1942-01-01 00:00:00,9999-12-31 23:59:59,Attignat,,,fr:departement:01@1860-07-01,3260,0
@@ -30,11 +33,14 @@ fr:commune:01028@1942-01-01,01028,1942-01-01 00:00:00,9999-12-31 23:59:59,Banein
 fr:commune:01029@1942-01-01,01029,1942-01-01 00:00:00,9999-12-31 23:59:59,Beaupont,,,fr:departement:01@1860-07-01,677,0
 fr:commune:01030@1942-01-01,01030,1942-01-01 00:00:00,9999-12-31 23:59:59,Beauregard,,,fr:departement:01@1860-07-01,876,0
 fr:commune:01031@1942-01-01,01031,1942-01-01 00:00:00,1957-08-21 23:59:59,Bélignat,fr:commune:01031@1957-08-22,,fr:departement:01@1860-07-01,NULL,100
+fr:commune:01031@1957-08-22,01031,1957-08-22 00:00:00,9999-12-31 23:59:59,Bellignat,,fr:commune:01031@1942-01-01,fr:departement:01@1860-07-01,3646,0
 fr:commune:01032@1942-01-01,01032,1942-01-01 00:00:00,9999-12-31 23:59:59,Béligneux,,,fr:departement:01@1860-07-01,3304,0
 fr:commune:01033@1942-01-01,01033,1942-01-01 00:00:00,1956-10-18 23:59:59,Bellegarde,fr:commune:01033@1956-10-19,,fr:departement:01@1860-07-01,NULL,100
+fr:commune:01033@1956-10-19,01033,1956-10-19 00:00:00,9999-12-31 23:59:59,Bellegarde-sur-Valserine,,fr:commune:01018@1942-01-01;fr:commune:01033@1942-01-01;fr:commune:01126@1942-01-01,fr:departement:01@1860-07-01,11644,0
 fr:commune:01034@1942-01-01,01034,1942-01-01 00:00:00,9999-12-31 23:59:59,Belley,,,fr:departement:01@1860-07-01,8909,0
 fr:commune:01035@1942-01-01,01035,1942-01-01 00:00:00,9999-12-31 23:59:59,Belleydoux,,,fr:departement:01@1860-07-01,327,0
 fr:commune:01036@1942-01-01,01036,1942-01-01 00:00:00,1974-10-31 23:59:59,Belmont,fr:commune:01036@1974-11-01,,fr:departement:01@1860-07-01,NULL,110
+fr:commune:01036@1974-11-01,01036,1974-11-01 00:00:00,9999-12-31 23:59:59,Belmont-Luthézieu,,fr:commune:01036@1942-01-01;fr:commune:01226@1942-01-01,fr:departement:01@1860-07-01,526,0
 fr:commune:01037@1942-01-01,01037,1942-01-01 00:00:00,9999-12-31 23:59:59,Bénonces,,,fr:departement:01@1860-07-01,276,0
 fr:commune:01038@1942-01-01,01038,1942-01-01 00:00:00,9999-12-31 23:59:59,Bény,,,fr:departement:01@1860-07-01,718,0
 fr:commune:01039@1942-01-01,01039,1942-01-01 00:00:00,9999-12-31 23:59:59,Béon,,,fr:departement:01@1860-07-01,457,0
@@ -52,6 +58,7 @@ fr:commune:01050@1942-01-01,01050,1942-01-01 00:00:00,9999-12-31 23:59:59,Boisse
 fr:commune:01051@1942-01-01,01051,1942-01-01 00:00:00,9999-12-31 23:59:59,Bolozon,,,fr:departement:01@1860-07-01,90,0
 fr:commune:01052@1942-01-01,01052,1942-01-01 00:00:00,9999-12-31 23:59:59,Bouligneux,,,fr:departement:01@1860-07-01,304,0
 fr:commune:01053@1942-01-01,01053,1942-01-01 00:00:00,1955-03-30 23:59:59,Bourg,fr:commune:01053@1955-03-31,,fr:departement:01@1860-07-01,NULL,100
+fr:commune:01053@1955-03-31,01053,1955-03-31 00:00:00,9999-12-31 23:59:59,Bourg-en-Bresse,,fr:commune:01053@1942-01-01,fr:departement:01@1860-07-01,40490,0
 fr:commune:01054@1942-01-01,01054,1942-01-01 00:00:00,9999-12-31 23:59:59,Bourg-Saint-Christophe,,,fr:departement:01@1860-07-01,1258,0
 fr:commune:01055@1942-01-01,01055,1942-01-01 00:00:00,1972-12-31 23:59:59,Bouvent,fr:commune:01283@1942-01-01,,fr:departement:01@1860-07-01,NULL,310
 fr:commune:01056@1942-01-01,01056,1942-01-01 00:00:00,9999-12-31 23:59:59,Boyeux-Saint-Jérôme,,,fr:departement:01@1860-07-01,365,0
@@ -76,9 +83,12 @@ fr:commune:01074@1942-01-01,01074,1942-01-01 00:00:00,9999-12-31 23:59:59,Chalam
 fr:commune:01075@1942-01-01,01075,1942-01-01 00:00:00,9999-12-31 23:59:59,Chaleins,,,fr:departement:01@1860-07-01,1199,0
 fr:commune:01076@1942-01-01,01076,1942-01-01 00:00:00,9999-12-31 23:59:59,Chaley,,,fr:departement:01@1860-07-01,146,0
 fr:commune:01077@1942-01-01,01077,1942-01-01 00:00:00,2006-07-08 23:59:59,Challes,fr:commune:01077@2006-07-09,,fr:departement:01@1860-07-01,NULL,100
+fr:commune:01077@2006-07-09,01077,2006-07-09 00:00:00,9999-12-31 23:59:59,Challes-la-Montagne,,fr:commune:01077@1942-01-01,fr:departement:01@1860-07-01,181,0
 fr:commune:01078@1942-01-01,01078,1942-01-01 00:00:00,9999-12-31 23:59:59,Challex,,,fr:departement:01@1860-07-01,1385,0
 fr:commune:01079@1942-01-01,01079,1942-01-01 00:00:00,1956-10-18 23:59:59,Champagne,fr:commune:01079@1956-10-19,,fr:departement:01@1860-07-01,NULL,100
+fr:commune:01079@1956-10-19,01079,1956-10-19 00:00:00,9999-12-31 23:59:59,Champagne-en-Valromey,,fr:commune:01079@1942-01-01;fr:commune:01217@1942-01-01;fr:commune:01287@1942-01-01,fr:departement:01@1860-07-01,805,0
 fr:commune:01080@1942-01-01,01080,1942-01-01 00:00:00,2015-12-31 23:59:59,Champdor,fr:commune:01080@2016-01-01,,fr:departement:01@1860-07-01,452,331
+fr:commune:01080@2016-01-01,01080,2016-01-01 00:00:00,9999-12-31 23:59:59,Champdor-Corcelles,,fr:commune:01080@1942-01-01;fr:commune:01119@1942-01-01,fr:departement:01@1860-07-01,679,341
 fr:commune:01081@1942-01-01,01081,1942-01-01 00:00:00,9999-12-31 23:59:59,Champfromier,,,fr:departement:01@1860-07-01,713,0
 fr:commune:01082@1942-01-01,01082,1942-01-01 00:00:00,9999-12-31 23:59:59,Chanay,,,fr:departement:01@1860-07-01,646,0
 fr:commune:01083@1942-01-01,01083,1942-01-01 00:00:00,9999-12-31 23:59:59,Chaneins,,,fr:departement:01@1860-07-01,847,0
@@ -87,14 +97,4 @@ fr:commune:01085@1942-01-01,01085,1942-01-01 00:00:00,9999-12-31 23:59:59,La Cha
 fr:commune:01086@1942-01-01,01086,1942-01-01 00:00:00,1973-12-31 23:59:59,Charancin,fr:commune:01414@1942-01-01,,fr:departement:01@1860-07-01,NULL,330
 fr:commune:01087@1942-01-01,01087,1942-01-01 00:00:00,9999-12-31 23:59:59,Charix,,,fr:departement:01@1860-07-01,284,0
 fr:commune:01088@1942-01-01,01088,1942-01-01 00:00:00,1991-03-20 23:59:59,Charnoz,fr:commune:01088@1991-03-21,,fr:departement:01@1860-07-01,NULL,100
-fr:commune:01089@1942-01-01,01089,1942-01-01 00:00:00,9999-12-31 23:59:59,Château-Gaillard,,,fr:departement:01@1860-07-01,1940,0
-fr:commune:01090@1942-01-01,01090,1942-01-01 00:00:00,9999-12-31 23:59:59,Châtenay,,,fr:departement:01@1860-07-01,339,0
-fr:commune:01091@1942-01-01,01091,1942-01-01 00:00:00,1973-10-31 23:59:59,Châtillon-de-Michaille,fr:commune:01091@1973-11-01,,fr:departement:01@1860-07-01,NULL,110
-fr:commune:01092@1942-01-01,01092,1942-01-01 00:00:00,9999-12-31 23:59:59,Châtillon-la-Palud,,,fr:departement:01@1860-07-01,1573,0
-fr:commune:01093@1942-01-01,01093,1942-01-01 00:00:00,9999-12-31 23:59:59,Châtillon-sur-Chalaronne,,,fr:departement:01@1860-07-01,4957,0
-fr:commune:01094@1942-01-01,01094,1942-01-01 00:00:00,9999-12-31 23:59:59,Chavannes-sur-Reyssouze,,,fr:departement:01@1860-07-01,732,0
-fr:commune:01095@1942-01-01,01095,1942-01-01 00:00:00,2016-12-31 23:59:59,Chavannes-sur-Suran,fr:commune:01095@2017-01-01,,fr:departement:01@1860-07-01,641,331
-fr:commune:01096@1942-01-01,01096,1942-01-01 00:00:00,9999-12-31 23:59:59,Chaveyriat,,,fr:departement:01@1860-07-01,975,0
-fr:commune:01097@1942-01-01,01097,1942-01-01 00:00:00,9999-12-31 23:59:59,Chavornay,,,fr:departement:01@1860-07-01,217,0
-fr:commune:01098@1942-01-01,01098,1942-01-01 00:00:00,2016-12-31 23:59:59,Chazey-Bons,fr:commune:01098@2017-01-01,,fr:departement:01@1860-07-01,831,331
-fr:commune:01099@1942-01-01,01099,1942-01-01 00:00:00,9999-12-31 23:59:59,Chazey-sur-Ain,,,fr:departement:01@1860-07-01,1544,0
+fr:commune:01088@1991-03-21,01088,1991-03-21 00:00:00,9999-12-31 23:59:59,Charnoz-sur-Ain,,fr:commune:01088@1942-01-01,fr:departement:01@1860-07-01,915,0

exports/epci/README.md

@@ -0,0 +1,54 @@
+# History of intercommunalities (EPCI)
+
+**The `epci.csv` file contains history for intercommunalities since 1999-01-01.**
+
+## Sources
+
+* https://fr.wikipedia.org/wiki/%C3%89tablissement_public_de_coop%C3%A9ration_intercommunale
+* https://www.collectivites-locales.gouv.fr/liste-et-composition-des-epci-a-fiscalite-propre
+
+
+## Columns
+
+* `id`: This is a [GeoID](https://github.com/etalab/geoids).
+* `siren`: The [9-digits code delivered by INSEE](https://www.data.gouv.fr/fr/datasets/base-sirene-des-entreprises-et-de-leurs-etablissements-siren-siret/).
+* `name`: The name of the intercommunality.
+* `acronym`: An optionnal known acronym or short name.
+* `kind`: The legal form of the intercommunality.
+* `taxmodel`: The tax model under which the intercommunality is.
+* `towns`: List of component towns `id`s separated by semicolons.
+* `start_date`: The effective start date for the current `id` using ISO format (`YYYY-MM-DD`).
+* `end_date`: The effective end date for the current `id` using ISO format (`YYYY-MM-DD`).
+* `end_reason`: The reason explaining the end of this intercommunality, see [Changes](#changes)
+* `successors`: List of `id`s separated by semicolons which are successors of the current `id`.
+* `ancestors`: List of `id`s separated by semicolons which are ancestors of the current `id`.
+* `population`: The legal population from last census.
+
+The `id` column is unique, the `siren` one is NOT.
+Arbitrarily, the far future end date has been set to `9999-12-31 23:59:59`.
+
+A new entry/line is registered when the intercommunality changed one of its:
+- `siren`
+- `name`
+- `taxmodel`
+- `kind`
+- component `towns`
+
+
+## Changes
+
+| code | Meaning |
+|------------|-----------------------------------------------------------------|
+| `inner` | This is an inner change, ie. perimeter changed, town renamed... |
+| `renamed` | The intercommunality changed its name |
+| `kind` | The intercommunality changed its legal form |
+| `taxmodel` | The intercommunality changed its tax model/system |
+| `removed` | The intercommunality does not exist anymore |
+| `merged` | The intercommunality has been merge with other(s) |
+
+
+## Format
+
+This is a regular CSV file with value separated by commas and a header line with previously described column names.
+
+You can have a look at the file using Github preview if you click on it.