diff --git a/README.md b/README.md index 70a507e..1234507 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,11 @@ # XSD files for the DK Hostmaster EPP service - + - [XSD file references](#xsd-file-references) - [XSD files](#xsd-files) - [XSD History](#xsd-history) + - [2.3](#23) - [2.2](#22) - [2.1](#21) - [2.0](#20) @@ -21,6 +22,7 @@ The current actively used XSD file is indicated in the [EPP service specification](https://github.com/DK-Hostmaster/epp-service-specification), this repository might contain changes not actively used by the service. + ## XSD file references Please note that the following files have been lifted from their respective RFCs. These files are copyrighted @@ -37,6 +39,7 @@ by their respective authors, please refer to the RFCs for more information. The following files are owned and copyright by DK Hostmaster A/S under the MIT License, please see the LICENSE file. + ## XSD files * dkhm-1.0.xsd, DK Hostmaster EPP extensions version 1.0 @@ -49,6 +52,7 @@ The following files are owned and copyright by DK Hostmaster A/S under the MIT L * dkhm-2.0.xsd, DK Hostmaster EPP extensions version 2.0 * dkhm-2.1.xsd, DK Hostmaster EPP extensions version 2.1 * dkhm-2.2.xsd, DK Hostmaster EPP extensions version 2.2 +* dkhm-2.3.xsd, DK Hostmaster EPP extensions version 2.3 * epp.xsd (a collection files for easier test, validation and maintenance, see below) The DK Hostmaster [EPP service specification](https://github.com/DK-Hostmaster/epp-service-specification) describes the use and contents of the files in more detail. @@ -61,45 +65,54 @@ $ xml --schema epp.xsd your_file.xml It will then either omit an error message or the success message: `your_file.xml validates`. + ## XSD History - + +### 2.3 + +- EPP Service version 2.3.X +- Introduction of `dkhm:url` for poll messages in relation to domain creation, where a URL is communicated, which can be presented to the end-user as part of the domain creation process. + + ### 2.2 -* Introduction of `dkhm:risk_assessment` for poll messages in relation to domain creation, where the risk assessment is communicated as part of the domain creation process. +- EPP Service version 2.3.X +- Introduction of `dkhm:risk_assessment` for poll messages in relation to domain creation, where the risk assessment is communicated as part of the domain creation process. - + ### 2.1 +- EPP Service version 2.3.X - **Warning!** This release includes a change to the standard XSD from [RFC:5730](https://tools.ietf.org/html/rfc5730), aligning the values for the password type. It has not been possible to get the patch applied using the XML Schema feature: `redefine` or `overwrite`. When this succeeds this change will have to be rolled-back. The change has been applied so the schema file conforms with the schema file used at DK Hostmaster A/S. - The DKHM Schema file has been updated to revision 2.1, the file does not contain any changes apart from the import, this file was created for a uniform communication in regard to revision numbers etc. - + ### 2.0 - Official release of changes proposed in revisions 1.5 and 1.6 - + ### 1.6 - Development use only - Introduction of `dkhm:requestedNsAdmin` for update host and create host - + ### 1.5 - Development use only - Introduction of `dkhm:mobilephone` on update contact - Introduction of `dkhm:secondaryEmail` on update contact - + ### 1.4 - EPP Service version 1.3.X - Introduction of `dkhm:pnumber` for production unit number information for create contact - + ### 1.3 - EPP Service version 1.2.X @@ -107,19 +120,19 @@ It will then either omit an error message or the success message: `your_file.xml - Introduction of `dkhm:contact_validated` for information for info contact - Introduction of `dkhm:registrant_validated` for information for create domain - + ### 1.2 - EPP Service version 1.1.X - Introduction of `dkhm:orderConfirmation` for create domain and support of [Pre-activation Service](#pre-activation-service) - + ### 1.1 - EPP Service version 1.0.9 - Introduction of `dkhm:domainAdvisory` for support of blocked status for create domain for blocked domain names - + ### 1.0 - EPP Service version 1.0.0 diff --git a/bin/cpanfile b/bin/cpanfile new file mode 100644 index 0000000..847a1e6 --- /dev/null +++ b/bin/cpanfile @@ -0,0 +1,4 @@ +requires 'XML::LibXML', '0'; +requires 'Try::Tiny', '0'; +requires 'XML::Twig', '0'; +requires 'XML::Validate::Xerces', '0'; diff --git a/bin/cpanfile.snapshot b/bin/cpanfile.snapshot new file mode 100644 index 0000000..3686129 --- /dev/null +++ b/bin/cpanfile.snapshot @@ -0,0 +1,529 @@ +# carton snapshot format: version 1.0 +DISTRIBUTIONS + Encode-Locale-1.05 + pathname: G/GA/GAAS/Encode-Locale-1.05.tar.gz + provides: + Encode::Locale 1.05 + requirements: + Encode 2 + Encode::Alias 0 + ExtUtils::MakeMaker 0 + perl 5.008 + File-Listing-6.04 + pathname: G/GA/GAAS/File-Listing-6.04.tar.gz + provides: + File::Listing 6.04 + File::Listing::apache 6.04 + File::Listing::dosftp 6.04 + File::Listing::netware 6.04 + File::Listing::unix 6.04 + File::Listing::vms 6.04 + requirements: + ExtUtils::MakeMaker 0 + HTTP::Date 6 + perl 5.006002 + File-Slurp-Tiny-0.004 + pathname: L/LE/LEONT/File-Slurp-Tiny-0.004.tar.gz + provides: + File::Slurp::Tiny 0.004 + requirements: + Carp 0 + Exporter 5.57 + ExtUtils::MakeMaker 0 + File::Spec::Functions 0 + FileHandle 0 + perl 5.008001 + strict 0 + warnings 0 + HTML-Parser-3.72 + pathname: G/GA/GAAS/HTML-Parser-3.72.tar.gz + provides: + HTML::Entities 3.69 + HTML::Filter 3.72 + HTML::HeadParser 3.71 + HTML::LinkExtor 3.69 + HTML::Parser 3.72 + HTML::PullParser 3.57 + HTML::TokeParser 3.69 + requirements: + ExtUtils::MakeMaker 0 + HTML::Tagset 3 + XSLoader 0 + perl 5.008 + HTML-Tagset-3.20 + pathname: P/PE/PETDANCE/HTML-Tagset-3.20.tar.gz + provides: + HTML::Tagset 3.20 + requirements: + ExtUtils::MakeMaker 0 + HTTP-Cookies-6.04 + pathname: O/OA/OALDERS/HTTP-Cookies-6.04.tar.gz + provides: + HTTP::Cookies 6.04 + HTTP::Cookies::Microsoft 6.04 + HTTP::Cookies::Netscape 6.04 + requirements: + Carp 0 + ExtUtils::MakeMaker 0 + HTTP::Date 6 + HTTP::Headers::Util 6 + HTTP::Request 0 + Time::Local 0 + locale 0 + perl 5.008001 + strict 0 + vars 0 + HTTP-Daemon-6.01 + pathname: G/GA/GAAS/HTTP-Daemon-6.01.tar.gz + provides: + HTTP::Daemon 6.01 + HTTP::Daemon::ClientConn 6.01 + requirements: + ExtUtils::MakeMaker 0 + HTTP::Date 6 + HTTP::Request 6 + HTTP::Response 6 + HTTP::Status 6 + IO::Socket 0 + LWP::MediaTypes 6 + Sys::Hostname 0 + perl 5.008001 + HTTP-Date-6.02 + pathname: G/GA/GAAS/HTTP-Date-6.02.tar.gz + provides: + HTTP::Date 6.02 + requirements: + ExtUtils::MakeMaker 0 + Time::Local 0 + perl 5.006002 + HTTP-Message-6.14 + pathname: O/OA/OALDERS/HTTP-Message-6.14.tar.gz + provides: + HTTP::Config 6.14 + HTTP::Headers 6.14 + HTTP::Headers::Auth 6.14 + HTTP::Headers::ETag 6.14 + HTTP::Headers::Util 6.14 + HTTP::Message 6.14 + HTTP::Request 6.14 + HTTP::Request::Common 6.14 + HTTP::Response 6.14 + HTTP::Status 6.14 + requirements: + Carp 0 + Compress::Raw::Zlib 0 + Encode 2.21 + Encode::Locale 1 + Exporter 5.57 + ExtUtils::MakeMaker 0 + HTTP::Date 6 + IO::Compress::Bzip2 2.021 + IO::Compress::Deflate 0 + IO::Compress::Gzip 0 + IO::HTML 0 + IO::Uncompress::Bunzip2 2.021 + IO::Uncompress::Gunzip 0 + IO::Uncompress::Inflate 0 + IO::Uncompress::RawInflate 0 + LWP::MediaTypes 6 + MIME::Base64 2.1 + MIME::QuotedPrint 0 + Storable 0 + URI 1.10 + base 0 + perl 5.008001 + strict 0 + warnings 0 + HTTP-Negotiate-6.01 + pathname: G/GA/GAAS/HTTP-Negotiate-6.01.tar.gz + provides: + HTTP::Negotiate 6.01 + requirements: + ExtUtils::MakeMaker 0 + HTTP::Headers 6 + perl 5.008001 + IO-HTML-1.001 + pathname: C/CJ/CJM/IO-HTML-1.001.tar.gz + provides: + IO::HTML 1.001 + requirements: + Carp 0 + Encode 2.10 + Exporter 5.57 + ExtUtils::MakeMaker 6.30 + LWP-MediaTypes-6.02 + pathname: G/GA/GAAS/LWP-MediaTypes-6.02.tar.gz + provides: + LWP::MediaTypes 6.02 + requirements: + ExtUtils::MakeMaker 0 + perl 5.006002 + Log-Trace-1.070 + pathname: B/BB/BBC/Log-Trace-1.070.tar.gz + provides: + Log::Trace 1.070 + requirements: + ExtUtils::MakeMaker 0 + Test::More 0 + Net-HTTP-6.17 + pathname: O/OA/OALDERS/Net-HTTP-6.17.tar.gz + provides: + Net::HTTP 6.17 + Net::HTTP::Methods 6.17 + Net::HTTP::NB 6.17 + Net::HTTPS 6.17 + requirements: + Carp 0 + Compress::Raw::Zlib 0 + ExtUtils::MakeMaker 0 + IO::Socket::INET 0 + IO::Uncompress::Gunzip 0 + URI 0 + base 0 + perl 5.006002 + strict 0 + vars 0 + warnings 0 + Test-Assertions-1.054 + pathname: B/BB/BBC/Test-Assertions-1.054.tar.gz + provides: + Test::Assertions 1.054 + Test::Assertions::TestScript 1.018 + requirements: + Carp 0 + ExtUtils::MakeMaker 0 + Getopt::Long 0 + Log::Trace 0 + Tree-DAG_Node-1.29 + pathname: R/RS/RSAVAGE/Tree-DAG_Node-1.29.tgz + provides: + Tree::DAG_Node 1.29 + requirements: + ExtUtils::MakeMaker 0 + File::Slurp::Tiny 0.003 + strict 0 + warnings 0 + Try-Tiny-0.30 + pathname: E/ET/ETHER/Try-Tiny-0.30.tar.gz + provides: + Try::Tiny 0.30 + requirements: + Carp 0 + Exporter 5.57 + ExtUtils::MakeMaker 0 + constant 0 + perl 5.006 + strict 0 + warnings 0 + URI-1.72 + pathname: E/ET/ETHER/URI-1.72.tar.gz + provides: + URI 1.72 + URI::Escape 3.31 + URI::Heuristic 4.20 + URI::IRI 1.72 + URI::QueryParam 1.72 + URI::Split 1.72 + URI::URL 5.04 + URI::WithBase 2.20 + URI::data 1.72 + URI::file 4.21 + URI::file::Base 1.72 + URI::file::FAT 1.72 + URI::file::Mac 1.72 + URI::file::OS2 1.72 + URI::file::QNX 1.72 + URI::file::Unix 1.72 + URI::file::Win32 1.72 + URI::ftp 1.72 + URI::gopher 1.72 + URI::http 1.72 + URI::https 1.72 + URI::ldap 1.72 + URI::ldapi 1.72 + URI::ldaps 1.72 + URI::mailto 1.72 + URI::mms 1.72 + URI::news 1.72 + URI::nntp 1.72 + URI::pop 1.72 + URI::rlogin 1.72 + URI::rsync 1.72 + URI::rtsp 1.72 + URI::rtspu 1.72 + URI::sftp 1.72 + URI::sip 1.72 + URI::sips 1.72 + URI::snews 1.72 + URI::ssh 1.72 + URI::telnet 1.72 + URI::tn3270 1.72 + URI::urn 1.72 + URI::urn::isbn 1.72 + URI::urn::oid 1.72 + requirements: + Carp 0 + Cwd 0 + Data::Dumper 0 + Encode 0 + Exporter 5.57 + ExtUtils::MakeMaker 0 + MIME::Base64 2 + Net::Domain 0 + Scalar::Util 0 + constant 0 + integer 0 + overload 0 + parent 0 + perl 5.008001 + strict 0 + utf8 0 + warnings 0 + WWW-RobotRules-6.02 + pathname: G/GA/GAAS/WWW-RobotRules-6.02.tar.gz + provides: + WWW::RobotRules 6.02 + WWW::RobotRules::AnyDBM_File 6.00 + WWW::RobotRules::InCore 6.02 + requirements: + AnyDBM_File 0 + ExtUtils::MakeMaker 0 + Fcntl 0 + URI 1.10 + perl 5.008001 + XML-Filter-BufferText-1.01 + pathname: R/RB/RBERJON/XML-Filter-BufferText-1.01.tar.gz + provides: + XML::Filter::BufferText 1.01 + requirements: + ExtUtils::MakeMaker 0 + Test::More 0.40 + XML::SAX 0.04 + XML::SAX::Base 1.03 + XML-LibXML-2.0132 + pathname: S/SH/SHLOMIF/XML-LibXML-2.0132.tar.gz + provides: + XML::LibXML 2.0132 + XML::LibXML::Attr 2.0132 + XML::LibXML::AttributeHash 2.0132 + XML::LibXML::Boolean 2.0132 + XML::LibXML::CDATASection 2.0132 + XML::LibXML::Comment 2.0132 + XML::LibXML::Common 2.0132 + XML::LibXML::Devel 2.0132 + XML::LibXML::Document 2.0132 + XML::LibXML::DocumentFragment 2.0132 + XML::LibXML::Dtd 2.0132 + XML::LibXML::Element 2.0132 + XML::LibXML::ErrNo 2.0132 + XML::LibXML::Error 2.0132 + XML::LibXML::InputCallback 2.0132 + XML::LibXML::Literal 2.0132 + XML::LibXML::NamedNodeMap 2.0132 + XML::LibXML::Namespace 2.0132 + XML::LibXML::Node 2.0132 + XML::LibXML::NodeList 2.0132 + XML::LibXML::Number 2.0132 + XML::LibXML::PI 2.0132 + XML::LibXML::Pattern 2.0132 + XML::LibXML::Reader 2.0132 + XML::LibXML::RegExp 2.0132 + XML::LibXML::RelaxNG 2.0132 + XML::LibXML::SAX 2.0132 + XML::LibXML::SAX::AttributeNode 2.0132 + XML::LibXML::SAX::Builder 2.0132 + XML::LibXML::SAX::Generator 2.0132 + XML::LibXML::SAX::Parser 2.0132 + XML::LibXML::Schema 2.0132 + XML::LibXML::Text 2.0132 + XML::LibXML::XPathContext 2.0132 + XML::LibXML::XPathExpression 2.0132 + XML::LibXML::_SAXParser 2.0132 + requirements: + ExtUtils::MakeMaker 0 + Test::More 0 + XML::NamespaceSupport 1.07 + XML::SAX 0.11 + XML::SAX::Base 0 + XML::SAX::Exception 0 + base 0 + parent 0 + perl 5.008 + strict 0 + vars 0 + warnings 0 + XML-NamespaceSupport-1.12 + pathname: P/PE/PERIGRIN/XML-NamespaceSupport-1.12.tar.gz + provides: + XML::NamespaceSupport 1.12 + requirements: + ExtUtils::MakeMaker 6.17 + constant 0 + perl 5.006 + strict 0 + vars 0 + warnings 0 + XML-Parser-2.44 + pathname: T/TO/TODDR/XML-Parser-2.44.tar.gz + provides: + XML::Parser 2.44 + XML::Parser::Expat 2.44 + XML::Parser::Style::Debug undef + XML::Parser::Style::Objects undef + XML::Parser::Style::Stream undef + XML::Parser::Style::Subs undef + XML::Parser::Style::Tree undef + requirements: + ExtUtils::MakeMaker 0 + LWP::UserAgent 0 + Test::More 0 + perl 5.00405 + XML-SAX-0.99 + pathname: G/GR/GRANTM/XML-SAX-0.99.tar.gz + provides: + XML::SAX 0.99 + XML::SAX::DocumentLocator undef + XML::SAX::ParserFactory 1.01 + XML::SAX::PurePerl 0.99 + XML::SAX::PurePerl::DebugHandler undef + XML::SAX::PurePerl::Exception undef + XML::SAX::PurePerl::Productions undef + XML::SAX::PurePerl::Reader undef + XML::SAX::PurePerl::Reader::Stream undef + XML::SAX::PurePerl::Reader::String undef + XML::SAX::PurePerl::Reader::URI undef + requirements: + ExtUtils::MakeMaker 0 + File::Temp 0 + XML::NamespaceSupport 0.03 + XML::SAX::Base 1.05 + XML-SAX-Base-1.09 + pathname: G/GR/GRANTM/XML-SAX-Base-1.09.tar.gz + provides: + XML::SAX::Base 1.09 + XML::SAX::Base::NoHandler 1.09 + XML::SAX::Exception 1.09 + requirements: + ExtUtils::MakeMaker 0 + perl 5.008 + XML-Twig-3.52 + pathname: M/MI/MIROD/XML-Twig-3.52.tar.gz + provides: + XML::Twig 3.52 + XML::Twig::Elt 3.52 + XML::Twig::Entity 3.52 + XML::Twig::Entity_list 3.52 + XML::Twig::Notation 3.52 + XML::Twig::Notation_list 3.52 + XML::Twig::XPath 0.02 + XML::Twig::XPath::Attribute 0.02 + XML::Twig::XPath::Elt 0.02 + XML::Twig::XPath::Namespace 0.02 + requirements: + ExtUtils::MakeMaker 0 + XML::Parser 2.23 + XML-Validate-1.025 + pathname: B/BB/BBC/XML-Validate-1.025.tar.gz + provides: + XML::Validate 1.025 + XML::Validate::Base 1.009 + XML::Validate::LibXML 1.020 + XML::Validate::MSXML 1.018 + XML::Validate::Xerces 1.021 + XML::Validate::Xerces::ErrorHandler 1.021 + requirements: + ExtUtils::MakeMaker 0 + Log::Trace 0 + Test::Assertions 0 + Test::More 0 + XML-Validator-Schema-1.10 + pathname: S/SA/SAMTREGAR/XML-Validator-Schema-1.10.tar.gz + provides: + XML::Validator::Schema 1.10 + XML::Validator::Schema::AllModelNode undef + XML::Validator::Schema::Attribute undef + XML::Validator::Schema::AttributeLibrary undef + XML::Validator::Schema::AttributeNode undef + XML::Validator::Schema::ChoiceModelNode undef + XML::Validator::Schema::ComplexTypeNode undef + XML::Validator::Schema::ElementLibrary undef + XML::Validator::Schema::ElementNode undef + XML::Validator::Schema::ElementRefNode undef + XML::Validator::Schema::Library undef + XML::Validator::Schema::ModelNode undef + XML::Validator::Schema::Node undef + XML::Validator::Schema::Parser undef + XML::Validator::Schema::RootNode undef + XML::Validator::Schema::SequenceModelNode undef + XML::Validator::Schema::SimpleType undef + XML::Validator::Schema::SimpleTypeNode undef + XML::Validator::Schema::TypeLibrary undef + XML::Validator::Schema::UnionModelNode undef + XML::Validator::Schema::Util undef + requirements: + Carp 0 + ExtUtils::MakeMaker 0 + Test::More 0.47 + Tree::DAG_Node 0 + XML::Filter::BufferText 0 + XML::SAX 0.12 + libwww-perl-6.31 + pathname: E/ET/ETHER/libwww-perl-6.31.tar.gz + provides: + LWP 6.31 + LWP::Authen::Basic 6.31 + LWP::Authen::Digest 6.31 + LWP::Authen::Ntlm 6.31 + LWP::ConnCache 6.31 + LWP::Debug 6.31 + LWP::Debug::TraceHTTP 6.31 + LWP::DebugFile 6.31 + LWP::MemberMixin 6.31 + LWP::Protocol 6.31 + LWP::Protocol::cpan 6.31 + LWP::Protocol::data 6.31 + LWP::Protocol::file 6.31 + LWP::Protocol::ftp 6.31 + LWP::Protocol::gopher 6.31 + LWP::Protocol::http 6.31 + LWP::Protocol::loopback 6.31 + LWP::Protocol::mailto 6.31 + LWP::Protocol::nntp 6.31 + LWP::Protocol::nogo 6.31 + LWP::RobotUA 6.31 + LWP::Simple 6.31 + LWP::UserAgent 6.31 + libwww::perl undef + requirements: + Digest::MD5 0 + Encode 2.12 + Encode::Locale 0 + ExtUtils::MakeMaker 0 + File::Copy 0 + File::Listing 6 + Getopt::Long 0 + HTML::Entities 0 + HTML::HeadParser 0 + HTTP::Cookies 6 + HTTP::Daemon 6 + HTTP::Date 6 + HTTP::Negotiate 6 + HTTP::Request 6 + HTTP::Request::Common 6 + HTTP::Response 6 + HTTP::Status 6 + IO::Select 0 + IO::Socket 0 + LWP::MediaTypes 6 + MIME::Base64 2.1 + Net::FTP 2.58 + Net::HTTP 6.07 + Scalar::Util 0 + Try::Tiny 0 + URI 1.10 + URI::Escape 0 + WWW::RobotRules 6 + base 0 + perl 5.008001 + strict 0 + warnings 0 diff --git a/bin/example.xml b/bin/example.xml new file mode 100644 index 0000000..2c5860f --- /dev/null +++ b/bin/example.xml @@ -0,0 +1,23 @@ + + + + + ClientX + foo-BAR2 + bar-FOO2bar-FOO2bar-FOO2 + + 1.0 + en + + + urn:ietf:params:xml:ns:obj1 + urn:ietf:params:xml:ns:obj2 + urn:ietf:params:xml:ns:obj3 + + http://custom/obj1ext-1.0 + + + + ABC-12345 + + diff --git a/bin/example2.pl b/bin/example2.pl new file mode 100644 index 0000000..a805b39 --- /dev/null +++ b/bin/example2.pl @@ -0,0 +1,26 @@ +#!/usr/bin/env perl + +use strict; +use warnings; + +use XML::SAX::ParserFactory; +use XML::Validator::Schema; + +my $xml_file = $ARGV[0]; +my $xsd_file = $ARGV[1]; + +# +# create a new validator object, using foo.xsd +# +my $validator = XML::Validator::Schema->new(file => $xsd_file); + +# +# create a SAX parser and assign the validator as a Handler +# +my $parser = XML::SAX::ParserFactory->parser(Handler => $validator); + +# +# validate foo.xml against foo.xsd +# +eval { $parser->parse_uri($xml_file) }; +die "File failed validation: $@" if $@; diff --git a/bin/example2.xml b/bin/example2.xml new file mode 100644 index 0000000..68b084d --- /dev/null +++ b/bin/example2.xml @@ -0,0 +1,4 @@ + + +2 + diff --git a/bin/local/bin/lwp-download b/bin/local/bin/lwp-download new file mode 100755 index 0000000..c35c47e --- /dev/null +++ b/bin/local/bin/lwp-download @@ -0,0 +1,335 @@ +#!/Users/jonasbn/perl5/perlbrew/perls/perl-5.22.3/bin/perl + +=head1 NAME + +lwp-download - Fetch large files from the web + +=head1 SYNOPSIS + + lwp-download [-a] [-s] [] + + Options: + + -a save the file in ASCII mode + -s use HTTP headers to guess output filename + +=head1 DESCRIPTION + +The B program will save the file at I to a local +file. + +If I is not specified, then the current directory is +assumed. + +If I is a directory, then the last segment of the path of the +I is appended to form a local filename. If the I path ends with +slash the name "index" is used. With the B<-s> option pick up the last segment +of the filename from server provided sources like the Content-Disposition +header or any redirect URLs. A file extension to match the server reported +Content-Type might also be appended. If a file with the produced filename +already exists, then B will prompt before it overwrites and will +fail if its standard input is not a terminal. This form of invocation will +also fail is no acceptable filename can be derived from the sources mentioned +above. + +If I is not a directory, then it is simply used as the +path to save into. If the file already exists it's overwritten. + +The I program is implemented using the I +library. It is better suited to down load big files than the +I program because it does not store the file in memory. +Another benefit is that it will keep you updated about its progress +and that you don't have much options to worry about. + +Use the C<-a> option to save the file in text (ASCII) mode. Might +make a difference on DOSish systems. + +=head1 EXAMPLE + +Fetch the newest and greatest perl version: + + $ lwp-download http://www.perl.com/CPAN/src/latest.tar.gz + Saving to 'latest.tar.gz'... + 11.4 MB received in 8 seconds (1.43 MB/sec) + +=head1 AUTHOR + +Gisle Aas + +=cut + +#' get emacs out of quote mode + +use strict; +use warnings; + +use LWP::UserAgent (); +use LWP::MediaTypes qw(guess_media_type media_suffix); +use URI (); +use HTTP::Date (); +use Encode; +use Encode::Locale; +use Getopt::Long qw(HelpMessage :config gnu_getopt no_ignore_case auto_help); + +my $progname = $0; +$progname =~ s,.*/,,; # only basename left in progname +$progname =~ s,.*\\,, if $^O eq "MSWin32"; +$progname =~ s/\.\w*$//; # strip extension if any + +my %opt; +GetOptions( + 'a' => \$opt{a}, + 's' => \$opt{s} +) or HelpMessage(); + +my $url = URI->new(decode(locale => shift) || HelpMessage()); +my $argfile = encode(locale_fs => decode(locale => shift)); +HelpMessage() if defined($argfile) && !length($argfile); + +my $ua = LWP::UserAgent->new( + agent => "lwp-download/$LWP::UserAgent::VERSION ", + keep_alive => 1, + env_proxy => 1, +); + +my $file; # name of file we download into +my $length; # total number of bytes to download +my $flength; # formatted length +my $size = 0; # number of bytes received +my $start_t; # start time of download +my $last_dur; # time of last callback + +my $shown = 0; # have we called the show() function yet + +$SIG{INT} = sub { die "Interrupted\n"; }; + +$| = 1; # autoflush + +my $res = $ua->request( + HTTP::Request->new(GET => $url), + sub { + unless (defined $file) { + my $res = $_[1]; + + my $directory; + if (defined $argfile && -d $argfile) { + ($directory, $argfile) = ($argfile, undef); + } + + unless (defined $argfile) { + + # find a suitable name to use + $file = $opt{s} && $res->filename; + + # if this fails we try to make something from the URL + unless ($file) { + $file = ($url->path_segments)[-1]; + if (!defined($file) || !length($file)) { + $file = "index"; + my $suffix = media_suffix($res->content_type); + $file .= ".$suffix" if $suffix; + } + elsif ($url->scheme eq 'ftp' + || $file =~ /\.t[bg]z$/ + || $file =~ /\.tar(\.(Z|gz|bz2?))?$/) + { + # leave the filename as it was + } + else { + my $ct = guess_media_type($file); + unless ($ct eq $res->content_type) { + + # need a better suffix for this type + my $suffix = media_suffix($res->content_type); + $file .= ".$suffix" if $suffix; + } + } + } + + # validate that we don't have a harmful filename now. The server + # might try to trick us into doing something bad. + if (!length($file) + || $file + =~ s/([^a-zA-Z0-9_\.\-\+\~])/sprintf "\\x%02x", ord($1)/ge + || $file =~ /^\./) + { + die + "Will not save <$url> as \"$file\".\nPlease override file name on the command line.\n"; + } + + if (defined $directory) { + require File::Spec; + $file = File::Spec->catfile($directory, $file); + } + + # Check if the file is already present + if (-l $file) { + die + "Will not save <$url> to link \"$file\".\nPlease override file name on the command line.\n"; + } + elsif (-f _) { + die + "Will not save <$url> as \"$file\" without verification.\nEither run from terminal or override file name on the command line.\n" + unless -t; + $shown = 1; + print "Overwrite $file? [y] "; + my $ans = ; + unless (defined($ans) && $ans =~ /^y?\n/) { + if (defined $ans) { + print "Ok, aborting.\n"; + } + else { + print "\nAborting.\n"; + } + exit 1; + } + $shown = 0; + } + elsif (-e _) { + die "Will not save <$url> as \"$file\". Path exists.\n"; + } + else { + print "Saving to '$file'...\n"; + use Fcntl qw(O_WRONLY O_EXCL O_CREAT); + sysopen(FILE, $file, O_WRONLY | O_EXCL | O_CREAT) + || die "Can't open $file: $!"; + } + } + else { + $file = $argfile; + } + unless (fileno(FILE)) { + open(FILE, ">", $file) || die "Can't open $file: $!\n"; + } + binmode FILE unless $opt{a}; + $length = $res->content_length; + $flength = fbytes($length) if defined $length; + $start_t = time; + $last_dur = 0; + } + + print FILE $_[0] or die "Can't write to $file: $!\n"; + $size += length($_[0]); + + if (defined $length) { + my $dur = time - $start_t; + if ($dur != $last_dur) { # don't update too often + $last_dur = $dur; + my $perc = $size / $length; + my $speed; + $speed = fbytes($size / $dur) . "/sec" if $dur > 3; + my $secs_left = fduration($dur / $perc - $dur); + $perc = int($perc * 100); + my $show = "$perc% of $flength"; + $show .= " (at $speed, $secs_left remaining)" if $speed; + show($show, 1); + } + } + else { + show(fbytes($size) . " received"); + } + } +); + +if (fileno(FILE)) { + close(FILE) || die "Can't write to $file: $!\n"; + + show(""); # clear text + print "\r"; + print fbytes($size); + print " of ", fbytes($length) if defined($length) && $length != $size; + print " received"; + my $dur = time - $start_t; + if ($dur) { + my $speed = fbytes($size / $dur) . "/sec"; + print " in ", fduration($dur), " ($speed)"; + } + print "\n"; + + if (my $mtime = $res->last_modified) { + utime time, $mtime, $file; + } + + if ($res->header("X-Died") || !$res->is_success) { + if (my $died = $res->header("X-Died")) { + print "$died\n"; + } + if (-t) { + print "Transfer aborted. Delete $file? [n] "; + my $ans = ; + if (defined($ans) && $ans =~ /^y\n/) { + unlink($file) && print "Deleted.\n"; + } + elsif ($length > $size) { + print "Truncated file kept: ", fbytes($length - $size), + " missing\n"; + } + else { + print "File kept.\n"; + } + exit 1; + } + else { + print "Transfer aborted, $file kept\n"; + } + } + exit 0; +} + +# Did not manage to create any file +print "\n" if $shown; +if (my $xdied = $res->header("X-Died")) { + print "$progname: Aborted\n$xdied\n"; +} +else { + print "$progname: ", $res->status_line, "\n"; +} +exit 1; + + +sub fbytes { + my $n = int(shift); + if ($n >= 1024 * 1024) { + return sprintf "%.3g MB", $n / (1024.0 * 1024); + } + elsif ($n >= 1024) { + return sprintf "%.3g KB", $n / 1024.0; + } + else { + return "$n bytes"; + } +} + +sub fduration { + use integer; + my $secs = int(shift); + my $hours = $secs / (60 * 60); + $secs -= $hours * 60 * 60; + my $mins = $secs / 60; + $secs %= 60; + if ($hours) { + return "$hours hours $mins minutes"; + } + elsif ($mins >= 2) { + return "$mins minutes"; + } + else { + $secs += $mins * 60; + return "$secs seconds"; + } +} + + +BEGIN { + my @ani = qw(- \ | /); + my $ani = 0; + + sub show { + my ($mess, $show_ani) = @_; + print "\r$mess" . (" " x (75 - length $mess)); + my $msg = $show_ani ? $ani[$ani++]. "\b" : ' '; + print $msg; + $ani %= @ani; + $shown++; + } +} diff --git a/bin/local/bin/lwp-dump b/bin/local/bin/lwp-dump new file mode 100755 index 0000000..f1381c6 --- /dev/null +++ b/bin/local/bin/lwp-dump @@ -0,0 +1,113 @@ +#!/Users/jonasbn/perl5/perlbrew/perls/perl-5.22.3/bin/perl + +use strict; +use warnings; +use LWP::UserAgent (); +use Getopt::Long qw(GetOptions); +use Encode; +use Encode::Locale; + +GetOptions(\my %opt, 'parse-head', 'max-length=n', 'keep-client-headers', + 'method=s', 'agent=s', 'request',) + || usage(); + +my $url = shift || usage(); +@ARGV && usage(); + +sub usage { + (my $progname = $0) =~ s,.*/,,; + die <<"EOT"; +Usage: $progname [options] + +Recognized options are: + --agent + --keep-client-headers + --max-length + --method + --parse-head + --request + +EOT +} + +my $ua = LWP::UserAgent->new( + parse_head => $opt{'parse-head'} || 0, + keep_alive => 1, + env_proxy => 1, + agent => $opt{agent} || "lwp-dump/$LWP::UserAgent::VERSION ", +); + +my $req = HTTP::Request->new($opt{method} || 'GET' => decode(locale => $url)); +my $res = $ua->simple_request($req); +$res->remove_header(grep /^Client-/, $res->header_field_names) + unless $opt{'keep-client-headers'} + or ($res->header("Client-Warning") || "") eq "Internal response"; + +if ($opt{request}) { + $res->request->dump; + print "\n"; +} + +$res->dump(maxlength => $opt{'max-length'}); + +__END__ + +=head1 NAME + +lwp-dump - See what headers and content is returned for a URL + +=head1 SYNOPSIS + +B [ I ] I + +=head1 DESCRIPTION + +The B program will get the resource identified by the URL and then +dump the response object to STDOUT. This will display the headers returned and +the initial part of the content, escaped so that it's safe to display even +binary content. The escapes syntax used is the same as for Perl's double +quoted strings. If there is no content the string "(no content)" is shown in +its place. + +The following options are recognized: + +=over + +=item B<--agent> I + +Override the user agent string passed to the server. + +=item B<--keep-client-headers> + +LWP internally generate various C headers that are stripped by +B in order to show the headers exactly as the server provided them. +This option will suppress this. + +=item B<--max-length> I + +How much of the content to show. The default is 512. Set this +to 0 for unlimited. + +If the content is longer then the string is chopped at the +limit and the string "...\n(### more bytes not shown)" +appended. + +=item B<--method> I + +Use the given method for the request instead of the default "GET". + +=item B<--parse-head> + +By default B will not try to initialize headers by looking at the +head section of HTML documents. This option enables this. This corresponds to +L. + +=item B<--request> + +Also dump the request sent. + +=back + +=head1 SEE ALSO + +L, L, L diff --git a/bin/local/bin/lwp-mirror b/bin/local/bin/lwp-mirror new file mode 100755 index 0000000..ca360f9 --- /dev/null +++ b/bin/local/bin/lwp-mirror @@ -0,0 +1,103 @@ +#!/Users/jonasbn/perl5/perlbrew/perls/perl-5.22.3/bin/perl + +# Simple mirror utility using LWP + +=head1 NAME + +lwp-mirror - Simple mirror utility + +=head1 SYNOPSIS + + lwp-mirror [-v] [-t timeout] + +=head1 DESCRIPTION + +This program can be used to mirror a document from a WWW server. The +document is only transferred if the remote copy is newer than the local +copy. If the local copy is newer nothing happens. + +Use the C<-v> option to print the version number of this program. + +The timeout value specified with the C<-t> option. The timeout value +is the time that the program will wait for response from the remote +server before it fails. The default unit for the timeout value is +seconds. You might append "m" or "h" to the timeout value to make it +minutes or hours, respectively. + +Because this program is implemented using the LWP library, it only +supports the protocols that LWP supports. + +=head1 SEE ALSO + +L, L + +=head1 AUTHOR + +Gisle Aas + +=cut + +use strict; +use warnings; +use LWP::Simple qw(mirror is_success status_message $ua); +use Getopt::Long qw(GetOptions); +use Encode; +use Encode::Locale; + +my $progname = $0; +$progname =~ s,.*/,,; # use basename only +$progname =~ s/\.\w*$//; #strip extension if any + +my %opts; +unless (GetOptions(\%opts, 'h', 'v', 't=i')) { + usage(); +} + +if ($opts{v}) { + require LWP; + my $DISTNAME = 'libwww-perl-' . $LWP::VERSION; + die <<"EOT"; +This is lwp-mirror version $LWP::Simple::VERSION ($DISTNAME) + +Copyright 1995-1999, Gisle Aas. + +This program is free software; you can redistribute it and/or +modify it under the same terms as Perl itself. +EOT +} + +my $url = decode(locale => shift) or usage(); +my $file = encode(locale_fs => decode(locale => shift)) or usage(); +usage() if $opts{h} or @ARGV; + +if ($opts{t}) { + if ($opts{t} =~ /^(\d+)([smh])?/) { + my $timeout = $1; + $timeout *= 60 if ($2 eq "m"); + $timeout *= 3600 if ($2 eq "h"); + $ua->timeout($timeout); + } + else { + die "$progname: Illegal timeout value!\n"; + } +} + +my $rc = mirror($url, $file); + +if ($rc == 304) { + print STDERR "$progname: $file is up to date\n"; +} +elsif (!is_success($rc)) { + print STDERR "$progname: $rc ", status_message($rc), " ($url)\n"; + exit 1; +} +exit; + + +sub usage { + die <<"EOT"; +Usage: $progname [-options] + -v print version number of program + -t Set timeout value +EOT +} diff --git a/bin/local/bin/lwp-request b/bin/local/bin/lwp-request new file mode 100755 index 0000000..e79988d --- /dev/null +++ b/bin/local/bin/lwp-request @@ -0,0 +1,561 @@ +#!/Users/jonasbn/perl5/perlbrew/perls/perl-5.22.3/bin/perl + +# Simple user agent using LWP library. + +=head1 NAME + +lwp-request, GET, POST, HEAD - Simple command line user agent + +=head1 SYNOPSIS + +B [B<-afPuUsSedvhx>] [B<-m> I] [B<-b> I] [B<-t> I] + [B<-i> I] [B<-c> I] + [B<-C> I] [B<-p> I] [B<-o> I] I... + +=head1 DESCRIPTION + +This program can be used to send requests to WWW servers and your +local file system. The request content for POST and PUT +methods is read from stdin. The content of the response is printed on +stdout. Error messages are printed on stderr. The program returns a +status value indicating the number of URLs that failed. + +The options are: + +=over 4 + +=item -m + +Set which method to use for the request. If this option is not used, +then the method is derived from the name of the program. + +=item -f + +Force request through, even if the program believes that the method is +illegal. The server might reject the request eventually. + +=item -b + +This URI will be used as the base URI for resolving all relative URIs +given as argument. + +=item -t + +Set the timeout value for the requests. The timeout is the amount of +time that the program will wait for a response from the remote server +before it fails. The default unit for the timeout value is seconds. +You might append "m" or "h" to the timeout value to make it minutes or +hours, respectively. The default timeout is '3m', i.e. 3 minutes. + +=item -i