Skip to content
SocNetV SocNetV
News

web crawler

5 posts with the tag “web crawler”

SocNetV v3.1 Released

Dimitris

SocNetV v3.1 Screenshot

Description

The Social Network Visualizer project announces a new version of our favorite SNA application. SocNetV v3.1 is the first Qt6-only release and brings many improvements and bug fixes.

What’s New in v3.1

  • Improved Performance: Faster large file loading and quicker responsiveness with large nets (greater than 20,000 edges), while having a reduced memory footprint.
  • Bug Fixes: Edge filtering now works as intended.

Availability

The new version is available for Windows, macOS, and Linux. Go to the SocNetV Downloads page to get it.

SocNetV v3.0 Released

Dimitris

SocNetV v3.0 is out now! It includes several performance improvements and new features to make social network analysis easier.

SocNetV v3.0 Screenshot

Description

The Social Network Visualizer project has released a brand-new version of our favorite SNA application. SocNetV v3.0 brings improved speed and responsiveness, bug fixes, and new command line parameters.

What’s New in v3.0

  • Improved Performance: Faster computations, as the Graph now runs on a separate thread.
  • Hardware-Accelerated Rendering: This version is the first to support hardware-accelerated (OpenGL) rendering of the networks on the canvas. This feature is enabled by default but can be disabled in Settings.
  • Web Crawler Enhancements: The Web Crawler has been improved to test for OpenSSL support in the OS, with user hints provided if OpenSSL is missing. Additionally, the delay between requests has been fixed.
  • Bug Fixes: A critical bug in weighted network centrality computations has been resolved (see #123).

Special Note

To run SocNetV v3.0 AppImage on Fedora 34 (which uses Wayland by default), use the following command:

env GDK_BACKEND=x11 ./SocNetV-3.0-dev-x86_64.AppImage

For Fedora/openSUSE users, you can get a proper package from our OBS repos.

Availability

The new version is available for Windows, macOS, and Linux. Go to the SocNetV Downloads page to get it.

SocNetV v2.5 released with lots of new features and new looks!

Dimitris

SocNetV v2.5 Screenshot

Description

The Social Network Visualizer project has released a brand new version of our favorite social network analysis and visualization software. SocNetV v2.5, codenamed “maniac”, brings new features and improvements and is now available for Windows, macOS, and Linux. Go to the SocNetV Downloads page and get it!

New Features and Improvements

  • Prominence Score Distribution: SocNetV now automatically computes the prominence scores distribution and displays it in a mini chart in the right sidebar. You can choose the chart type (line, area, or bars) from the app settings. A larger version of this chart is also exported in every HTML prominence report.

    Prominence Distribution

  • Search for and Select Multiple Nodes by Numbers, Labels, Scores: You can now search for multiple nodes by their number, label, or prominence score. Matched nodes are highlighted.

    Search Nodes by Prominence Score

  • Lattice Network Generator: Create regular lattices of arbitrary dimensions. Define the lattice dimension d and the length/size l of each dimension. You can also define a custom neighborhood size n to set the distance within which neighbors on the lattice will be connected.

    Lattice Network Generator

  • Edge Dichotomization: For weighted networks, you can now dichotomize the edges according to a threshold. Edges with weights above the threshold will become binary, converting the network to a non-valued graph displayed in a new relation.

  • Custom Icons in Nodes: SocNetV now allows you to use custom images (e.g., PNG, JPEG, SVG) as node icons. You can choose the desired image from Settings -> Nodes. Any image file from your filesystem can be used as a custom node icon in your network.

    Custom Node Icons

    Additionally, SocNetV offers many built-in icons (e.g., person, computer, bug).

    Node Icons Settings

  • Export Network to Any Image Format: Export your network to any image format supported by Qt (e.g., PNG, JPG, BMP, TIFF).

  • Much Improved PDF Exporting: You can now select orientation, quality, and DPI when exporting the network to a PDF file.

  • Uniform UI Theme: A high-quality UI theme inspired by Material Design, providing a uniform look and feel across all operating systems.

  • New Control Panel Toolbox Menu: Network Auto Create: You can now automatically create a network with one click in the new toolbox menu. Random networks, famous datasets, and web crawlers are supported.

  • Improved Web Crawler: The built-in web crawler now runs faster and allows you to exclude/include social network links, parent, and child links.

    Web Crawler

  • Support for Floating Point Edge Weights: SocNetV now supports floating point (double) edge weights in Adjacency, GraphML, Pajek, edge lists, GML, etc. This fixes a bug with non-integer edge weights when importing Adjacency files.

    This version also brings speed optimization and improved memory management.

Availability

SocNetV v2.5 is now available for Windows, macOS, and Linux. Go to the SocNetV Downloads page to get it!

SocNetV 2.4 Released

Dimitris

SocNetV 2.4 Screenshot

Description

The Social Network Visualizer project released today a brand new version of our favorite social network analysis and visualization software application. SocNetV version 2.4, released on Feb 28, is a major upgrade bringing lots of new features. The new version is available for Windows, macOS, and Linux from the Downloads page.

New Features and Improvements

  • Kamada-Kawai Layout Model: This model treats the network as a dynamic system where actors are connected by “springs.” The layout optimizes the graph based on spring energy, minimizing the imbalance between desired and actual distances.

    Kamada-Kawai Layout

  • Node Colors by Prominence Score Visualization: In this version, SocNetV can visualize the prominence score of each actor by changing their color. The color reflects how important the actor is in the network, with red indicating high prominence and blue indicating lower prominence.

    Node Color by Prominence

  • Reciprocated Edges in Directed Networks: SocNetV now displays reciprocated edges in a single line with two arrows instead of two separate lines, reducing memory consumption and making the visualization more appealing.

    Reciprocated Edges

  • Improved Memory Consumption: The new version optimizes memory usage when loading large networks, improving performance for graphs with more than 2,000 edges.

  • Web Crawler Improvements: The web crawler dialog and algorithm have been enhanced with new options, including the ability to specify link patterns to include/exclude and control the delay between requests.

    Web Crawler

  • Improved Statistics Panel: The statistics panel now uses simpler text to display statistics more efficiently. It also shows edge direction weights for reciprocated edges in directed networks.

  • Performance Options in Settings: New performance options in the Settings dialog allow you to enable/disable features of the Qt GraphicsView, such as anti-aliasing and edge highlighting. Disabling certain options can significantly boost performance for large networks.

    Performance Options

  • Improved UCINET Format Support: SocNetV now supports reading two-mode UCINET files formatted in fullmatrix. These files declare both NR (nodes) and NC (columns) variables.

  • Improved Stability: Many bugs have been fixed, and random crashes were resolved. Below are some of the bugs that were fixed:

    • #40: Wrong BC scores in weighted networks
    • #54: Incorrect edge weight after user confirmation
    • #38: Disabling isolate nodes not reflected in distance matrix report
    • #41: Incorrect variance in clustering coefficient
    • #33: Missing Radial/Level Layout by Eigenvector Centrality
    • #48: Eccentricity scores for isolated nodes
    • #34: No menu options for Node Size layout by Prominence index
    • #56: Edge offset issues when resizing nodes
    • #46: Crashes after disabling isolated nodes in distance matrix
    • #42: Improved progress dialogs
    • #44: Crash when searching after node removal
    • #51: Web crawler crashes fixed
    • #60: Incorrect file extension when saving famous datasets
    • #61: “Save As” issue with GraphML format
    • #62: EdgeList1 formatted UCINET files not recognized
    • #63: Two-mode UCINET files unsupported
    • #65: Crashes in HCA with ill-defined input matrices
    • #36: HCA crashes with isolated nodes
    • #49: Connectedness method issues
    • #47: Extra columns in adjacency matrix report

Availability

SocNetV v2.4 is now available for Windows, macOS, and Linux. Go to the SocNetV Downloads page to get it!

SocNetV v1.6 Released with a Working Web Crawler

Dimitris

The SocNetV project has just released its latest version 1.6. Binaries for Windows, Mac OS X, and Linux are available from the Downloads menu.

Revamped Web Crawler

The new version brings back the web crawler feature, which had been disabled in the 1.x series so far.

To start the web crawler:

  • Go to Network -> Web Crawler or press Shift+C.

A dialog will appear where you:

  • Enter the initial web page (seed).
  • Set the maximum number of nodes/pages (default is 600).
  • Choose the types of links to crawl: internal, external, or both. By default, the crawler processes both.

Web Crawler Dialog

How It Works

The new web crawler is significantly improved compared to the 0.x releases. It consists of two components:

  1. Spider: Visits the specified initial URL, downloads its HTML, and processes it.
  2. Parser: Scans the downloaded HTML for href links to internal or external pages and adds them to a queue of URLs (the “frontier”).

The spider and parser run on separate threads, ensuring faster execution.

As URLs are added to the queue, the spider visits them, downloads their HTML, and the parser extracts more links, continuing the cycle.

Crawler in Action

The process is multithreaded and completes within seconds, even for large sets like 1,000 URLs.

Results

The crawler creates a network of all visited webpages as nodes and their links as edges. By default, node sizes are proportional to their outDegree, making patterns visible immediately.

Crawler Output

From there, you can analyze the network using the SNA tools provided by SocNetV.

Analyzing the Crawled Network

Note: The parser searches for href links only in the body section of the HTML.

Explore and analyze networks effortlessly with the enhanced capabilities of SocNetV v1.6!