o Improve documentation for MASTER_SITES:n mechanism for both simple

and advanced usage cases
o Reviewer's offered lots of improvements

Reviewed by:	keramida, nik
This commit is contained in:
Mario Sergio Fujikawa Ferreira 2002-12-28 13:35:15 +00:00
parent 3625438fed
commit e7f2c1ecb6
Notes: svn2git 2020-12-08 03:00:23 +00:00
svn path=/head/; revision=15451

View file

@ -1939,46 +1939,645 @@ EXTRACT_ONLY= source.tar.gz</programlisting>
</note>
</sect2>
<sect2>
<title>Multiple distribution files from different sites</title>
<sect2 id="porting-master-sites-n">
<title>Multiple distribution files or patches from different
sites and subdirectories
(<literal>MASTER_SITES:n</literal>)</title>
<para>Some applications consist of multiple distribution files that
must be downloaded from a number of different sites. For example,
<application>Ghostscript</application> consists of the core of the
program, and then a large number of driver files that are used
depending on the user's printer. Some of these driver files are
supplied with the core, but many others must be downloaded from a
variety of different sites.</para>
<para>This section has information on the fetching mechanism
known as both <literal>MASTER_SITES:n</literal> and
<literal>MASTER_SITES_NN</literal>. We will refer to this
mechanism as <literal>MASTER_SITES:n</literal>
hereon.</para>
<para>To support this, each entry in <makevar>DISTFILES</makevar> may
be followed by a colon and a <quote>tag name</quote>. Each site
listed in <makevar>MASTER_SITES</makevar> is then followed by a
colon, and the tag that indicates which distribution files should be
downloaded from this site.</para>
<para>A little background first. OpenBSD has a neat feature
inside both <makevar>DISTFILES</makevar> and
<makevar>PATCHFILES</makevar> variables, both files and
patches can be postfixed with <literal>:n</literal>
identifiers where <literal>n</literal> both can be
<literal>[0-9]</literal> and denote a group designation.
For example:</para>
<para>For example, consider an application with the source split in to
<filename>source1.tar.gz</filename> and
<filename>source2.tar.gz</filename>, which must be downloaded from
two different sites. The port's <filename>Makefile</filename> would
include lines like this:</para>
<programlisting>DISTFILES= alpha:0 beta:1</programlisting>
<programlisting>MASTER_SITES= ftp://ftp.example1.com/:source1 \
ftp://ftp.example2.com/:source2
DISTFILES= source1.tar.gz:source1 \
source2.tar.gz:source2</programlisting>
<para>In OpenBSD, distribution file <filename>alpha</filename>
will be associated with variable
<makevar>MASTER_SITES0</makevar> instead of our common
<makevar>MASTER_SITES</makevar> and
<filename>beta</filename> with
<makevar>MASTER_SITES1</makevar>.</para>
<para>Multiple distribution files can have the same tag. Continuing
the previous example, suppose that there was a third distfile,
<filename>source3.tar.gz</filename>, that should be downloaded from
<hostid>ftp.example2.com</hostid>. The
<filename>Makefile</filename> would then be written like
this.</para>
<para>This is a very interesting feature which can decrease
that endless search for the correct download site.</para>
<programlisting>MASTER_SITES= ftp://ftp.example1.com/:source1 \
ftp://ftp.example2.com/:source2
DISTFILES= source1.tar.gz:source1 \
source2.tar.gz:source2 \
source3.tar.gz:source2</programlisting>
<para>Just picture 2 files in <makevar>DISTFILES</makevar> and
20 sites in <makevar>MASTER_SITES</makevar>, the sites slow
as hell where <filename>beta</filename> is carried by all
sites in <makevar>MASTER_SITES</makevar>, and
<filename>alpha</filename> can only be found in the 20th
site. It would be such a waste to check all of them if
maintainer knew this beforehand, would not it? Not a good
start for that lovely weekend!</para>
<para>Now that you got the picture, just imagine more
<makevar>DISTFILES</makevar> and more
<makevar>MASTER_SITES</makevar>. Surely our
<quote>distfiles survey meister</quote> would appreciate the
network strain relieve this would bring.</para>
<para>In the next sections, information will follow on the
FreeBSD implementation of this idea. We improved a bit on
OpenBSD's concept.</para>
<sect3>
<title>Simplified information</title>
<para>This section tells you how to quickly prepare fine
grained fetching of multiple distribution files and
patches from different sites and subdirectories. We
describe here a case of simplified
<literal>MASTER_SITES:n</literal> usage. This will be
sufficient for most scenarios. However, if you need
further information, you will have to refer to the next
section.</para>
<para>Some applications consist of multiple distribution
files that must be downloaded from a number of different
sites. For example,
<application>Ghostscript</application> consists of the
core of the program, and then a large number of driver
files that are used depending on the user's printer. Some
of these driver files are supplied with the core, but many
others must be downloaded from a variety of different
sites.</para>
<para>To support this, each entry in
<makevar>DISTFILES</makevar> may be followed by a colon
and a <quote>tag name</quote>. Each site listed in
<makevar>MASTER_SITES</makevar> is then followed by a
colon, and the tag that indicates which distribution files
should be downloaded from this site.</para>
<para>For example, consider an application with the source
split in two parts, <filename>source1.tar.gz</filename>
and <filename>source2.tar.gz</filename>, which must be
downloaded from two different sites. The port's
<filename>Makefile</filename> would include lines like
<xref
linkend="ports-master-sites-n-example-simple-use-one-file-per-site">.</para>
<example
id="ports-master-sites-n-example-simple-use-one-file-per-site">
<title>Simplified use of <literal>MASTER_SITES:n</literal>
with 1 file per site</title>
<programlisting>MASTER_SITES= ftp://ftp.example1.com/:source1 \
ftp://ftp.example2.com/:source2
DISTFILES= source1.tar.gz:source1 \
source2.tar.gz:source2</programlisting>
</example>
<para>Multiple distribution files can have the same tag.
Continuing the previous example, suppose that there was a
third distfile, <filename>source3.tar.gz</filename>, that
should be downloaded from
<hostid>ftp.example2.com</hostid>. The
<filename>Makefile</filename> would then be written like
<xref
linkend="ports-master-sites-n-example-simple-use-more-than-one-file-per-site">.</para>
<example
id="ports-master-sites-n-example-simple-use-more-than-one-file-per-site">
<title>Simplified use of <literal>MASTER_SITES:n</literal>
with more than 1 file per site</title>
<programlisting>MASTER_SITES= ftp://ftp.example1.com/:source1 \
ftp://ftp.example2.com/:source2
DISTFILES= source1.tar.gz:source1 \
source2.tar.gz:source2 \
source3.tar.gz:source2</programlisting>
</example>
</sect3>
<sect3>
<title>Detailed information</title>
<para>Okay, so the previous section example did not reflect
your needs? In this section we will explain in detail how
the fine grained fetching mechanism
<literal>MASTER_SITES:n</literal> works and how you can
modify your ports to use it.</para>
<orderedlist>
<listitem>
<para>Elements can be postfixed with
<literal>:<replaceable>n</replaceable></literal> where
<replaceable>n</replaceable> is
<literal>[^:,]+</literal>, i.e.,
<replaceable>n</replaceable> could conceptually be any
alphanumerical string but we will limit it to
<literal>[a-zA-Z_][0-9a-zA-Z_]+</literal> for
now.</para>
<para>Moreover, string matching is case sensitive;
i.e., <literal>n</literal> is different from
<literal>N</literal>.</para>
<para>However, the following words cannot be used for
postfixing purposes since they yield special meaning:
<literal>default</literal>, <literal>all</literal> and
<literal>ALL</literal> (they are used internally in
item <xref
linkend="porting-master-sites-n-what-changes-in-port-targets">).
Furthermore, <literal>DEFAULT</literal> is a special
purpose word (check item <xref
linkend="porting-master-sites-n-DEFAULT-group">).</para>
</listitem>
<listitem>
<para>Elements postfixed with <literal>:n</literal>
belong to the group <literal>n</literal>,
<literal>:m</literal> belong to group
<literal>m</literal> and so forth.</para>
</listitem>
<listitem id="porting-master-sites-n-DEFAULT-group">
<para>Elements without a postfix are groupless, i.e.,
they all belong to the special group
<literal>DEFAULT</literal>. If you postfix any
elements with <literal>DEFAULT</literal>, you are just
being reduntant unless you want to have an element
belonging to both <literal>DEFAULT</literal> and other
groups at the same time (check item <xref
linkend="porting-master-sites-n-comma-operator">).</para>
<para>The following examples are equivalent but the
first one is preferred:</para>
<programlisting>MASTER_SITES= alpha
MASTER_SITES= alpha:DEFAULT</programlisting>
</listitem>
<listitem>
<para>Groups are not exclusive, an element may belong to
several different groups at the same time and a group
can either have either several different elements or
none at all. Repeated elements within the same group
will be simply that, repeated elements.</para>
</listitem>
<listitem id="porting-master-sites-n-comma-operator">
<para>When you want an element to belong to several
groups at the same time, you can use the comma
operator (<literal>,</literal>).</para>
<para>Instead of repeating it several times, each time
with a different postfix, we can list several groups
at once in a single postfix. For instance,
<literal>:m,n,o</literal> marks an element that
belongs to group <literal>m</literal>,
<literal>n</literal> and <literal>o</literal>.</para>
<para>All the following examples are equivalent but the
last one is preferred:</para>
<programlisting>MASTER_SITES= alpha alpha:SOME_SITE
MASTER_SITES= alpha:DEFAULT alpha:SOME_SITE
MASTER_SITES= alpha:SOME_SITE,DEFAULT
MASTER_SITES= alpha:DEFAULT,SOME_SITE</programlisting>
</listitem>
<listitem>
<para>All sites within a given group are sorted
according to <makevar>MASTER_SORT_AWK</makevar>. All
groups within <makevar>MASTER_SITES</makevar> and
<makevar>PATCH_SITES</makevar> are sorted as
well.</para>
</listitem>
<listitem id="porting-master-sites-n-group-semantics">
<para>Group semantics can be used in any of the
following variables <makevar>MASTER_SITES</makevar>,
<makevar>PATCH_SITES</makevar>,
<makevar>MASTER_SITE_SUBDIR</makevar>,
<makevar>PATCH_SITE_SUBDIR</makevar>,
<makevar>DISTFILES</makevar>, and
<makevar>PATCHFILES</makevar> according to the
following syntax:</para>
<orderedlist>
<listitem>
<para>All <makevar>MASTER_SITES</makevar>,
<makevar>PATCH_SITES</makevar>,
<makevar>MASTER_SITE_SUBDIR</makevar> and
<makevar>PATCH_SITE_SUBDIR</makevar> elements must
be terminated with the forward slash
<literal>/</literal> character. If any elements
belong to any groups, the group postfix
<literal>:<replaceable>n</replaceable></literal>
must come right after the terminator
<literal>/</literal>. The
<literal>MASTER_SITES:n</literal> mechanism relies
on the existence of the terminator
<literal>/</literal> to avoid confusing elements
where a <literal>:n</literal> is a valid part of
the element with occurences where
<literal>:n</literal> denotes group
<literal>n</literal>. For compatibility purposes,
since the <literal>/</literal> terminator was not
required before in both
<makevar>MASTER_SITE_SUBDIR</makevar> and
<makevar>PATCH_SITE_SUBDIR</makevar> elements, if
the postfix immediate preceeding character is not
a <literal>/</literal> then <literal>:n</literal>
will be considered a valid part of the element
instead of a group postfix even if an element is
postfixed with <literal>:n</literal>. See both
<xref
linkend="ports-master-sites-n-example-detailed-use-master-site-subdir">
and <xref
linkend="ports-master-sites-n-example-detailed-use-complete-example-master-sites">.</para>
<example id="ports-master-sites-n-example-detailed-use-master-site-subdir">
<title>Detailed use of
<literal>MASTER_SITES:n</literal> in
<makevar>MASTER_SITE_SUBDIR</makevar></title>
<programlisting>MASTER_SITE_SUBDIR= old:n new/:NEW</programlisting>
<itemizedlist>
<listitem>
<para>Directories within group
<literal>DEFAULT</literal> -> old:n</para>
</listitem>
<listitem>
<para>Directories within group
<literal>NEW</literal> -> new</para>
</listitem>
</itemizedlist>
</example>
<example
id="ports-master-sites-n-example-detailed-use-complete-example-master-sites">
<title>Detailed use of
<literal>MASTER_SITES:n</literal> with comma
operator, multiple files, multiple sites and
multiple subdirectories</title>
<programlisting>MASTER_SITES= http://site1/%SUBDIR%/ http://site2/:DEFAULT \
http://site3/:group3 http://site4/:group4 \
http://site5/:group5 http://site6/:group6 \
http://site7/:DEFAULT,group6 \
http://site8/%SUBDIR%/:group6,group7 \
http://site9/:group8
DISTFILES= file1 file2:DEFAULT file3:group3 \
file4:group4,group5,group6 file5:grouping \
file6:group7
MASTER_SITE_SUBDIR= directory-trial:1 directory-n/:groupn \
directory-one/:group6,DEFAULT \
directory</programlisting>
<para>The previous example results in the
following fine grained fetching. Sites are
listed in the exact order they will be
used.</para>
<itemizedlist>
<listitem>
<para><filename>file1</filename> will be
fetched from</para>
<itemizedlist>
<listitem>
<para><makevar>MASTER_SITE_OVERRIDE</makevar></para>
</listitem>
<listitem>
<para>http://site1/directory/</para>
</listitem>
<listitem>
<para>http://site1/directory-one/</para>
</listitem>
<listitem>
<para>http://site1/directory-trial:1/</para>
</listitem>
<listitem>
<para>http://site2/</para>
</listitem>
<listitem>
<para>http://site7/</para>
</listitem>
<listitem>
<para><makevar>MASTER_SITE_BACKUP</makevar></para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para><filename>file2</filename> will be
fetched exactly as
<filename>file1</filename> since they
both belong to the same group</para>
<itemizedlist>
<listitem>
<para><makevar>MASTER_SITE_OVERRIDE</makevar></para>
</listitem>
<listitem>
<para>http://site1/directory/</para>
</listitem>
<listitem>
<para>http://site1/directory-one/</para>
</listitem>
<listitem>
<para>http://site1/directory-trial:1/</para>
</listitem>
<listitem>
<para>http://site2/</para>
</listitem>
<listitem>
<para>http://site7/</para>
</listitem>
<listitem>
<para><makevar>MASTER_SITE_BACKUP</makevar></para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para><filename>file3</filename> will be
fetched from</para>
<itemizedlist>
<listitem>
<para><makevar>MASTER_SITE_OVERRIDE</makevar></para>
</listitem>
<listitem>
<para>http://site3/</para>
</listitem>
<listitem>
<para><makevar>MASTER_SITE_BACKUP</makevar></para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para><filename>file4</filename> will be
fetched from</para>
<itemizedlist>
<listitem>
<para><makevar>MASTER_SITE_OVERRIDE</makevar></para>
</listitem>
<listitem>
<para>http://site4/</para>
</listitem>
<listitem>
<para>http://site5/</para>
</listitem>
<listitem>
<para>http://site6/</para>
</listitem>
<listitem>
<para>http://site7/</para>
</listitem>
<listitem>
<para>http://site8/directory-one/</para>
</listitem>
<listitem>
<para><makevar>MASTER_SITE_BACKUP</makevar></para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para><filename>file5</filename> will be
fetched from</para>
<itemizedlist>
<listitem>
<para><makevar>MASTER_SITE_OVERRIDE</makevar></para>
</listitem>
<listitem>
<para><makevar>MASTER_SITE_BACKUP</makevar></para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para><filename>file6</filename> will be
fetched from</para>
<itemizedlist>
<listitem>
<para><makevar>MASTER_SITE_OVERRIDE</makevar></para>
</listitem>
<listitem>
<para>http://site8/directory-one/</para>
</listitem>
<listitem>
<para><makevar>MASTER_SITE_BACKUP</makevar></para>
</listitem>
</itemizedlist>
</listitem>
</itemizedlist>
</example>
</listitem>
</orderedlist>
</listitem>
<listitem>
<para>How do I group one of the special variables from
<filename>bsd.sites.mk</filename>, e.g.,
<makevar>MASTER_SITE_SOURCEFORGE</makevar>?</para>
<para>See <xref
linkend="ports-master-sites-n-example-detailed-use-master-site-sourceforge">.</para>
<example
id="ports-master-sites-n-example-detailed-use-master-site-sourceforge">
<title>Detailed use of
<literal>MASTER_SITES:n</literal> with
<makevar>MASTER_SITE_SOURCEFORGE</makevar></title>
<programlisting>MASTER_SITES= http://site1/ ${MASTER_SITE_SOURCEFORGE:S/$/:sourceforge,TEST/}
DISTFILES= something.tar.gz:sourceforge</programlisting>
</example>
<para><filename>something.tar.gz</filename> will be
fetched from all sites within
<makevar>MASTER_SITE_SOURCEFORGE</makevar>.</para>
</listitem>
<listitem>
<para>How do I use this with <makevar>PATCH*</makevar>
variables?</para>
<para>All examples were done with
<makevar>MASTER*</makevar> variables but they work
exactly the same for <makevar>PATCH*</makevar> ones as
can be seen in <xref
linkend="ports-master-sites-n-example-detailed-use-patch-sites">.</para>
<example
id="ports-master-sites-n-example-detailed-use-patch-sites">
<title>Simplified use of
<literal>MASTER_SITES:n</literal> with
<makevar>PATCH_SITES</makevar>.</title>
<programlisting>PATCH_SITES= http://site1/ http://site2/:test
PATCHFILES= patch1:test</programlisting>
</example>
</listitem>
</orderedlist>
</sect3>
<sect3>
<title>What does change for ports? What does not?</title>
<orderedlist numeration="lowerroman">
<listitem>
<para>All current ports remain the same. The
<literal>MASTER_SITES:n</literal> feature code is only
activated if there are elements postfixed with
<literal>:<replaceable>n</replaceable></literal> like
elements according to the aforementioned syntax rules,
especially as shown in item <xref
linkend="porting-master-sites-n-group-semantics">.</para>
</listitem>
<listitem id="porting-master-sites-n-what-changes-in-port-targets">
<para>The port targets remain the same:
<maketarget>checksum</maketarget>,
<maketarget>makesum</maketarget>,
<maketarget>patch</maketarget>,
<maketarget>configure</maketarget>,
<maketarget>build</maketarget>, etc. With the obvious
exceptions of <maketarget>do-fetch</maketarget>,
<maketarget>fetch-list</maketarget>,
<maketarget>master-sites</maketarget> and
<maketarget>patch-sites</maketarget>.</para>
<itemizedlist>
<listitem>
<para><maketarget>do-fetch</maketarget>: deploys the
new grouping postfixed
<makevar>DISTFILES</makevar> and
<makevar>PATCHFILES</makevar> with their matching
group elements within both
<makevar>MASTER_SITES</makevar> and
<makevar>PATCH_SITES</makevar> which use matching
group elements within both
<makevar>MASTER_SITE_SUBDIR</makevar> and
<makevar>PATCH_SITE_SUBDIR</makevar>. Check <xref
linkend="ports-master-sites-n-example-detailed-use-complete-example-master-sites">.</para>
</listitem>
<listitem>
<para><maketarget>fetch-list</maketarget>: works
like old <maketarget>fetch-list</maketarget> with
the exception that it groups just like
<maketarget>do-fetch</maketarget>.</para>
</listitem>
<listitem>
<para><maketarget>master-sites</maketarget> and
<maketarget>patch-sites</maketarget>:
(incompatible with older versions) only return the
elements of group <literal>DEFAULT</literal>; in
fact, they execute targets
<maketarget>master-sites-default</maketarget> and
<maketarget>patch-sites-default</maketarget>
respectively.</para>
<para>Furthermore, using target either
<maketarget>master-sites-all</maketarget> or
<maketarget>patch-sites-all</maketarget> is
preferred to directly checking either
<maketarget>MASTER_SITES</maketarget> or
<maketarget>PATCH_SITES</maketarget>. Also,
directly checking is not guaranteed to work in any
future versions. Check item <xref
linkend="porting-master-sites-n-new-port-targets-master-sites-all">
for more information on these new port
targets.</para>
</listitem>
</itemizedlist>
</listitem>
<listitem>
<para>New port targets</para>
<orderedlist>
<listitem>
<para>There are
<maketarget>master-sites-<replaceable>n</replaceable></maketarget>
and
<maketarget>patch-sites-<replaceable>n</replaceable></maketarget>
targets which will list the elements of the
respective group <replaceable>n</replaceable>
within <makevar>MASTER_SITES</makevar> and
<makevar>PATCH_SITES</makevar> respectively. For
instance, both
<maketarget>master-sites-DEFAULT</maketarget> and
<maketarget>patch-sites-DEFAULT</maketarget> will
return the elements of group
<literal>DEFAULT</literal>,
<maketarget>master-sites-test</maketarget> and
<maketarget>patch-sites-test</maketarget> of group
<literal>test</literal>, and thereon.</para>
</listitem>
<listitem id="porting-master-sites-n-new-port-targets-master-sites-all">
<para>There are new targets
<maketarget>master-sites-all</maketarget> and
<maketarget>patch-sites-all</maketarget> which do
the work of the old
<maketarget>master-sites</maketarget> and
<maketarget>patch-sites</maketarget> ones. They
return the elements of all groups as if they all
belonged to the same group with the caveat that it
lists as many
<makevar>MASTER_SITE_BACKUP</makevar> and
<makevar>MASTER_SITE_OVERRIDE</makevar> as there
are groups defined within either
<makevar>DISTFILES</makevar> or
<makevar>PATCHFILES</makevar>; respectively for
<maketarget>master-sites-all</maketarget> and
<maketarget>patch-sites-all</maketarget>.</para>
</listitem>
</orderedlist>
</listitem>
</orderedlist>
</sect3>
</sect2>
<sect2>