Concurrent projects with downstream projects

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Concurrent projects with downstream projects

mhesidence
I have a projectX that runs concurrently on 2+ hosts.      I would like to split projectX so that the compile/link is separate from the unit tests.    Does a downsteam project run on the same host as the upsteam project?  All I see is that a downsteam project goes into the build queue so obviously that would be bad of the unit tests did not immidately follow the compile/link on the same host (each host is using local disk).

I'm thinking maybe the parameterized trigger plugin to start default with compile=true then launch itself again with unit_test=true, but how would I control hudson so it would use the same node?



Reply | Threaded
Open this post in threaded view
|

Re: Concurrent projects with downstream projects

Sami Tikka
The not recommended for a job to use build artifacts of another job by
copying them from the other job's workspace. This would mean the two
jobs need to execute on the same node, which might be non-optimal if
other nodes are free.

The recommended pattern goes like this:

0) Install Copy artifacts plugin

1) Configure the upstream job to "archive build artifacts" and list
files in the workspace that should be archived. Optional (but very
much recommended) is to also click on "Record fingerprints of files to
track usage" and "Fingerprint all archived artifacts".

2) Configure the downstream job to have a first build step "copy
artifacts from another project", then tell it which artifact you want
from the upstream project. The next build step can then call your
build script/ant/whatever and use the upstream's build artifact which
Hudson has served into the downsteram job's workspace. Optional (but
very much recommended)  is to click on "Record fingerprints of files
to track usage" and then list the artifacts that were copied from the
upstream job.

This way the build artifacts are stored in Hudson, there are no
synchronization problems with one job poking around the workspace of
another job and everything works even if jobs run on different nodes.

Fingerprinting helps you keep track of the builds which have produced
the artifacts and which builds have used them.

-- Sami

2010/9/3 mhesidence <[hidden email]>:

>
> I have a projectX that runs concurrently on 2+ hosts.      I would like to
> split projectX so that the compile/link is separate from the unit tests.
> Does a downsteam project run on the same host as the upsteam project?  All I
> see is that a downsteam project goes into the build queue so obviously that
> would be bad of the unit tests did not immidately follow the compile/link on
> the same host (each host is using local disk).
>
> I'm thinking maybe the parameterized trigger plugin to start default with
> compile=true then launch itself again with unit_test=true, but how would I
> control hudson so it would use the same node?
>
>
>
>
> --
> View this message in context: http://hudson.361315.n4.nabble.com/Concurrent-projects-with-downstream-projects-tp2525976p2525976.html
> Sent from the Hudson users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Concurrent projects with downstream projects

Les Mikesell
On 9/4/10 7:09 AM, Sami Tikka wrote:

> The not recommended for a job to use build artifacts of another job by
> copying them from the other job's workspace. This would mean the two
> jobs need to execute on the same node, which might be non-optimal if
> other nodes are free.
>
> The recommended pattern goes like this:
>
> 0) Install Copy artifacts plugin
>
> 1) Configure the upstream job to "archive build artifacts" and list
> files in the workspace that should be archived. Optional (but very
> much recommended) is to also click on "Record fingerprints of files to
> track usage" and "Fingerprint all archived artifacts".
>
> 2) Configure the downstream job to have a first build step "copy
> artifacts from another project", then tell it which artifact you want
> from the upstream project. The next build step can then call your
> build script/ant/whatever and use the upstream's build artifact which
> Hudson has served into the downsteram job's workspace. Optional (but
> very much recommended)  is to click on "Record fingerprints of files
> to track usage" and then list the artifacts that were copied from the
> upstream job.
>
> This way the build artifacts are stored in Hudson, there are no
> synchronization problems with one job poking around the workspace of
> another job and everything works even if jobs run on different nodes.
>
> Fingerprinting helps you keep track of the builds which have produced
> the artifacts and which builds have used them.

Can you expand on this a bit to describe how you might control versioning and
perhaps releases with this technique?   Would there be any way to specify that
the downstream build should use a particular version of an upstream component or
does everything have to track the latest builds?

--
   Les Mikesell
    [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Concurrent projects with downstream projects

Sami Tikka
2010/9/4 Les Mikesell <[hidden email]>:
>
> Can you expand on this a bit to describe how you might control versioning
> and perhaps releases with this technique?   Would there be any way to
> specify that the downstream build should use a particular version of an
> upstream component or does everything have to track the latest builds?

The Copy artifact build step allows you to specify the project and the
build from which to copy the artifacts. So it would be possible to
freeze the build number of a component for releasing.

The downside could be that this build number is not stored in version
control, only in Hudson job configuration. Of course, it is possible
to take the config.xml file from the job directory from the master and
store that in your verson control system.

There is also another way to do this: Do not use Copy artifact
plugin/build step. Use wget or curl or something similar in your
downstream build script to download artifacts of upstream jobs. If
your build script is stored in version control (and if it isn't,
you've got other big problems :), you URLs in the build script fix the
exact build numbers of upstream components.

The URLs are of the format
$HUDSON_URL/job/JOBNAME/BUILDNUMBER/artifacts/ARTIFACT

During development you would use "lastSuccessfulBuild" or
"lastStableBuild" in the URL and replace them with the exact build
number when you decide on the component build going into the release.

Third way to go about this is what I use at work:

Each component is built by its own Hudson job. The jobs all archive
their artifacts and fingerprint them.

Finally one master build picks up the artifacts from
"lastSuccessfulBuild" of each component job and creates a product
installer. The installer is archived as an artifact and fingerprinted.

The master build then triggers a number of test jobs, which pick up
the installer and test it. The test jobs create test reports in JUnit
xml format for publishing by Hudson and fingerprint the installer they
downloaded.

This means that we can open the master build job in Hudson and click
on the little card icon next to installer artifact and Hudson will
tell us which tests has this installer been through and have the tests
been successful. It can also tell us which were the component
artifacts that were used. (Hudson does this all by matching the
fingerprints (=MD5 checksums) of artifact files it sees in the builds
of various jobs.)

If we are ready to release (everything is done and all tests are
good), we click on the "Keep this build forever" button for the master
build and all the component builds that had artifacts in it. We write
description on the Hudson page of the master build saying this build
was released on date so and so.

Actually I have simplified the description a bit but maybe you got the idea.

-- Sami

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]