FOSSLight Dependency Scanner
FOSSLight Dependency Scanner is the tool that supports the analysis of dependencies for multiple package managers. It detects the manifest file of package managers automatically and analyzes the dependencies with using open source tools. Then, it generates the report file that contains OSS information of dependencies.
Supported Package Managers
Github Repository : https://github.com/fosslight/fosslight_dependency_scanner
License : Apache-2.0
Contents
📋 Prerequisite
Because we utilize the different open source software to analyze the dependencies of each package manager, you need to set up the below Prerequisite steps according to package manager to analyze.
Prerequisite for NPM
- Install the NPM License Checker to analyze the npm dependencies.
$ npm install -g license-checker
To install license-checker globally, ‘-g' option is required. If you do not have ‘sudo' access, then you can change default path to install global modules.
$ npm set prefix ~/.npm $ PATH=~/.npm/bin:$PATH
- Run the command to install the dependencies. (optional)
$ npm install
It can be skipped if the project meets any of the following cases.
- If the ‘package.json' file exists in the input directory, it will be executed automatically by FOSSLight Dependency Scanner. So you can skip it.
- If the ‘node_modules' directory already exists, you can run FOSSLight Dependency Scanner by setting the input directory to the path where node_modules is located.
Prerequisite for Gradle
- Add the License Gradle Plugin in build.gradle file.
plugins { id 'com.github.hierynomus.license' version '0.16.1' // If the gradle version is 6.x or lower, then add the '0.15.0' version instead of '0.16.1'. } downloadLicenses { includeProjectDependencies = true dependencyConfiguration = 'runtimeClasspath' // If the gradle version is 4.6 or lower, then add the 'runtime' instead of 'runtimeClasspath'. }
- Run the ‘downloadLicenses' task.
$ gradlew downloadLicenses
Prerequisite for Android (gradle)
If there is a ‘gradlew’ executable and a ‘build.gradle’ file in the input directory, FOSSLight Dependency Scanner will automatically add and execute the android-dependency-scanning plugin. So you can skip the prerequisites below.
- Add the android-dependency-scanning Plugin in build.gradle file.
buildscript { repositories { mavenCentral() } dependencies { classpath 'org.fosslight:android-dependency-scanning:1.0.0' } }
- Add the below line in build.gradle file in the app(your application name, default : app) directory. It must be added to the subline of the plugins block (or apply plugin:'com.android.application') to prevent an error from occurring.
apply plugin: 'org.fosslight'
- Run the ‘generateLicenseTxt' task.
$ gradlew generateLicenseTxt
Prerequisite for Pypi
- You can run this tool with virtual environment for separating the project dependencies from system global dependencies.
- If the ‘requirements.txt’ file is located in the input path, FOSSLight Dependency Scanner can automatically install and analyze the dependencies. So you can skip from the prerequisite step2 for Pypi.
- Install python3-venv.
$ sudo apt-get install python3-venv
- Create and activate the virtual environment
// virtualenv example $ virtualenv -p /usr/bin/python3.7 venv $ source venv/bin/activate // conda example $ conda create --name {venv name} $ conda activate {venv name}
- Install the packages that you use in the virtual environment.
- You can add activate, deactivate command with ‘-a', ‘-d' option to run FOSSLight Dependency Scanner.
Prerequisite for Maven
If the ‘pom.xml’ is located in the input directory, FOSSLight Dependency Scanner will automatically add and execute the license-maven-plugin. So you can skip the prerequisites below.
- Add the license-maven-plugin into pom.xml file.
- Run the license-maven-plugin task.
<project> ... <build> ... <plugins> ... <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>license-maven-plugin</artifactId> <version>2.0.0</version> <executions> <execution> <id>aggregate-download-licenses</id> <goals> <goal>aggregate-download-licenses</goal> </goals> </execution> </executions> </plugin> </plugins> ... </build> ... </project>
$ mvnw license:aggregate-download-licenses
Prerequisite for Pub
If the ‘flutter pub' command is not available in the environment where FOSSLight Dependency Scanner is running, please perform the following steps in advance in the environment where ‘flutter pub' is available.
- Run flutter_oss_licenses with below command. (optional)
$ flutter pub add dev:flutter_oss_licenses:'^2.0.1' $ flutter pub get $ flutter pub deps --json > tmp_deps.json $ flutter pub deps --no-dev -s compact > tmp_no_dev_deps.txt $ flutter pub run flutter_oss_licenses:generate.dart -o tmp_flutter_oss_licenses.json --json
- Run FOSSLight Dependency Scanner in the path where the file generated in the result of step 1 exists.
Prerequisite for Cocoapods
- Install the pod package through Podfile.
$ pod install
Prerequisite for Swift
- Create a github personal access token and use it with ‘-t' option when running the FOSSLight dependency scanner. It needs the Github API to get the license information of the github repository.
Please refer the github docs guide to create a token.
Prerequisite for Carthage
- Create ‘Cartfile.resolved' by running the package installation command.
$ carthage update
- Create a github personal access token and use it with ‘-t' option when running the FOSSLight dependency scanner. It needs the Github API to get the license information of the github repository.
Please refer the github docs guide to create a token.
Prerequisite for Go
FOSSLight Dependency Scanner only supports for go modules. It automatically executes the ‘go list -m all’ command to obtain a list of dependencies, and then collects the open source software information such as license and repository. Therefore, you can execute the ‘fosslight_dependency’ command directly without prerequisite step.
Prerequisite for Nuget
FOSSLight Dependency Scanner checks the package list through the packages.config file or obj/project.assets.json file in case of PackageReference, and then prints the OSS information of dependencies such as license and repository through nuget api. Therefore, you can execute the ‘fosslight_dependency’ command directly without prerequisite step.
Prerequisite for Helm
FOSSLight Dependency Scanner checks the package list and OSS information such as license and repository through the Chart.yaml and ‘helm dependency build’ command.Therefore, you can execute the ‘fosslight_dependency’ command directly without prerequisite step.
Prerequisite for Unity
FOSSLight Dependency Scanner checks the package list and OSS information such as license and repository through the Library/PackageManager/ProjectCache file and each package directory within the Library/PackageCache directory. Therefore, you can execute the ‘fosslight_dependency’ command in an environment where the files exist.
Prerequisite for Cargo
FOSSLight Dependency Scanner checks the package list and OSS information such as license and repository through the Cargo.toml and ‘cargo metadata’ command.Therefore, you can execute the ‘fosslight_dependency’ command directly without prerequisite step.
🎉 How to install
FOSSLight Dependency Scanner can be installed using pip3.
It is recommended to install in the python 3.7 + virtualenv environment.
$ pip install fosslight-dependency
🚀 How to run
You can run the FOSSLight Dependency Scanner with options based on your package manager.
$ fosslight_dependency [option] <arg>
Options
Optional
-h Print help message.
-v Print the version of the script.
-m <package_manager> Enter the package manager.
(npm, maven, gradle, pypi, pub, cocoapods, android, swift, carthage, go, nuget, helm, unity, cargo)
-p <input_path> Enter the path where the script will be run.
-e <exclude_path> Enter the path where the analysis will not be performed. (Pattern matching is available)
-o <output_path> Output path
(If you want to generate the specific file name, add the output path with file name.)
-f <format> [<format> ...] Output formats (excel, csv, opossum, yaml, spdx-tag, spdx-yaml, spdx-json, spdx-xml)
Multiple formats can be specified separated by space.
--graph-path <save_path> Enter the path where the graph image will be saved
(ex. /your/directory/path/filename.{pdf, jpg, png}) (recommend pdf extension)
--graph-size <width> <height> Enter the size of the graph image (The size unit is pixels)
--graph-path option is required
--direct Print the direct/transitive dependency type in comment.
Choice 'True' or 'False'. (default:True)
--notice Print the open source license notice text.
Required only for swift, carthage
-t <token> Enter the github personal access token.
Optional only for pypi
-a <activate_cmd> Virtual environment activate command(ex, 'conda activate (venv name)')
-d <deactivate_cmd> Virtual environment deactivate command(ex, 'conda deactivate')
Optional only for gradle, maven
-c <dir_name> Enter the customized build output directory name
-Default name : 'build' for gradle, 'target' for maven
Optional only for android
-n <app_name> Enter the application directory name where the plugin output file is located(default: app)
Tips to run
When you run the FOSSLight Dependency Scanner, the input path(‘-p' option) should be designated as the top directory of the project where the package manager's manifest file exists as above. The manifest file of each package manager is as follows:
- Npm : package.json
- Pypi : requirements.txt / setup.py / pyproject.toml
- Maven : pom.xml
- Gradle (Android) : build.gradle
- Pub : pubspec.yaml
- Cocoapods : Podfile
- Swift : Package.resolved
- Carthage : Cartfile.resolved
- Go : go.mod
- Nuget : packages.config / {project name}.csproj
- Helm : Chart.yaml
- Unity : Library/PackageManager/ProjectCache
- Cargo : Cargo.toml
- Android (gradle)
- If the module name is not the default ‘app', you must run it by specifying the module name with the ‘-n' option. (fosslight_dependency -n {module_name})
- Swift package manager
- Exceptionally, you can run "fosslight_dependency -m swift -t {token} command in the path where {Projectname}.xcodeproj file is located.
- Then it can find the ‘Package.resolved' file in {Projectname}.xcodeproj/project.xcworkspace/xcshareddata/swiftpm and run automatically.
- Unity
- You can run "fosslight_dependency -m unity" command in the path where ‘Library' folder is located.
📁 Result
$ tree
.
├── fosslight_report_dep_210503_0039.xlsx
├── fosslight_log_dep_210503_0039.txt
└── fosslight_opossum_dep_210503_0039.json
- fosslight_report_dep_[datetime].xlsx : FOSSLight Dependency Scanner result in spreadsheet format.
- fosslight_log_dep_[datetime].txt: The execution log.
- fosslight_opossum_dep_[datetime].json : FOSSLight Dependency Scanner result for OpossumUI (-f opossum)
- third_party_notice.txt : Created only when running with Unity, and collects and prints the third party notice for each package.
Graph Network Creation Result
# $ fosslight_dependency -p /project/path --graph-path ~/temp/graph.png --graph-size 1000 1000
$ cd ~/temp
$ tree
.
└── graph.png
- Saved an image of the dependency graph using the Depends On section from the fosslight_report_dep_[datetime].xlsx file
Result Contents
It prints the OSS information based on manifest file(package.json, pom.xml) of dependencies (including transitive dependencies). For a unique OSS name, OSS name is printed such as (package_manager):(oss name) or (group id):(artifact id).
Package manager | OSS Name | Download Location | Homepage |
---|---|---|---|
Npm | npm:(oss name) | Priority1. repository in package.json Priority2. npmjs.com/package/(oss name)/v/(oss version) | npmjs.com/package/(oss name) |
Pypi | pypi:(oss name) | pypi.org/project/(oss name)/(version) | homepage in (pip show) information |
Maven & Gradle & Android | (group_id):(artifact_id) | mvnrepository.com/artifact/(group id)/(artifact id)/(version) | mvnrepository.com/artifact/(group id)/(artifact id) |
Pub | pub:(oss name) | pub.dev/packages/(oss name)/versions/(version) | homepage in (pub information) |
Cocoapods | cocoapods:(oss name) | source in (pod spec information) | cocoapods.org/pods/(oss name) |
Swift | swift:(oss name) | repositoryURL in Package.resolved | repositoryURL in Package.resolved |
Carthage | carthage:(oss name) | github repository in Cartfile.resolved | github repository in Cartfile.resolved |
Go | go:(oss name) | pkg.go.dev/(oss name)@(oss version) | repository in pkg.go.dev/(oss name)@(oss version) |
Nuget | nuget:(oss name) | Priority1. repository in nuget.org/packages/(oss name)/(oss version) Priority2. projectUrl in nuget.org/packages/(oss name)/(oss version) Priority3. nuget.org/packages/(oss name)/(oss version) | nuget.org/packages/(oss name) |
Helm | helm:(oss name) | first url of sources in (Chart.yaml) | home in (Chart.yaml) |
Unity | (oss name) | url in repository in ProjectCache | url in repository in ProjectCache |
Cargo | cargo:(oss name) | repository of the package in the result file for ‘cargo metadata' | crates.io/crates/(oss name) |
- The printed download location of npm, maven, gradle may be different from the url of actual package if installed through the local path or local repository (not distributed in npmjs.com or mvnrepository).
- For Helm, the dependencies of each dependency are not currently supported. And it obtains the OSS information of each dependency from the Chart.yaml file information in the .tgz file downloaded in the charts/ directory after executing the ‘helm dependency build’ command. Therefore, if information such as License or Homepage is missing in Chart.yaml, the information cannot be obtained, so the user needs to manually check and supplement it.
🧐 How it works
FOSSLight Dependency Scanner utilizes the open source software for analyzing each package manager dependencies. We choose the open source software for each package manager that shows not only the direct dependencies but also the transitive dependencies including the information of dependencies such as oss name, oss version and license name.
Each package manager uses the results of the following software:
- NPM : NPM License Checker
- Pypi : pip-licenses
- Gradle : License Gradle Plugin
- Maven : license-maven-plugin
- Pub : flutter_oss_licenses
- Android(gradle) : android-dependency-scanning
Because we utilizes the different open source software to analyze the dependencies of each package manager, you need to set up the Prerequisite steps according to package manager to analyze.
👀 Package Support Level
Language/ Project | Package Manager | Manifest file | Direct dependencies | Transitive dependencies | Relationship of dependencies (Dependencies of each dependency) |
---|---|---|---|---|---|
Javascript | Npm | package.json | O | O | O |
Java | Gradle | build.gradle | O | O | O |
Maven | pom.xml | O | O | O | |
Java (Android) | Gradle | build.gradle | O | O | O |
ObjC, Swift (iOS) | Cocoapods | Podfile.lock | O | O | O |
Carthage | Cartfile.resolved | O | O | X | |
Swift (iOS) | Swift | Package.resolved | O | O | O |
Dart, Flutter | Pub | pubspec.yaml | O | O | O |
Go | Go | go.mod | O | O | O |
Python | Pypi | requirements.txt, setup.py, pyproject.toml | O | O | O |
.NET | Nuget | packages.config, obj/project.assets.json | O | O | O |
Kubernetes | Helm | Chart.yaml | O | X | X |
Unity | Unity | Library/PackageManager/ProjectCache | O | O | X |
Rust | Cargo | Cargo.toml | O | O | O |