Courses
File and Data Management
Navigating and Searching through Local and Remote Filesystems
Sciebo/NextCloud/Owncloud Folders as a Remote Filesystem

Sciebo/NextCloud/Owncloud Folders as a Remote Filesystem

Author
Dr. Nicholas Del Grosso
Open in JupyterLite Download Exercises

Cloud storage services like Sciebo, NextCloud, and Owncloud offer convenient solutions for storing and accessing large datasets, and they are offered free of charge to all researchers in the NRW region! This section focuses on integrating these services into your data analysis pipeline, demonstrating how to interact with these cloud storage platforms programmatically to manage your neuroscience research data efficiently.

These systems include a “WebDAV” interface for interacting with them as a remote filesystem. In this section, we’ll use a WebdavFileSystem fsspec class to connect!

If you haven’t made a Sciebo account yet, here are some links:

Setup

Import Libraries 

from pathlib import Path
from webdav4.fsspec import WebdavFileSystem
import os
from webdav4.fsspec import WebdavFileSystem
import fsspec  # webdav4 should be installed
import requests
from dotenv import load_dotenv
load_dotenv(override=True)
env = os.environ

Section 1: Read-Only, Public Datasets

Accessing public datasets is a common requirement in neuroscience research. In Sciebo, you can make one instantly by creating a “Public Link” to a folder.

  • URL: https://uni-bonn.sciebo.de/public.php/webdav,
  • Username: The code at the end of the public link.
  • Passsword: Whatever password you made for the link, or an empty string if no password.

This section provides insights into connecting with read-only, public datasets using WebDAV-based file systems.

Exercises

Example: List all the files in the root directory of this public Sciebo Link: https://uni-bonn.sciebo.de/s/BftPyWxYk48EiTm 

fs = WebdavFileSystem("https://uni-bonn.sciebo.de/public.php/webdav", auth=("BftPyWxYk48EiTm", ""))
fs.ls("/", detail=False)
['README.md', 'data', 'scripts']

Exercise: What files are in the “scripts” folder?

Solution 
fs.ls("scripts", detail=False)
['scripts/analyze_data.py']

Exercise: How many files are in the “data/processed” folder?

Solution 
fs.ls("data/processed", detail=False)
['data/processed/file1.md', 'data/processed/file2.md']

Section 2: Remote Filesystems that Require Authentication: Basic Secret Handling

Dealing with remote filesystems often involves authentication to ensure data security. This section introduces basic secret handling techniques to securely connect to protected data resources. We’ll cover how to safely store and use authentication credentials, a critical skill for maintaining the integrity and confidentiality of sensitive neuroscience data

Pick one person in your group to create a shared folder for the group to upload data to. That person should:

  • Using Sciebo in the web browser, create a new, empty folder in their Sciebo account.
  • Create a new text file in that folder called “README.txt”, and put a special message inside the file.
  • For each team member to share the code with:
    • Share the folder by creating a “public link”, giving the
      • name of the team member as the link’s name,
      • permission to download/view/upload/edit,
      • a password,
      • and set the link to expire in the near-ish future.
    • Using the Zoom Chat, give each person the share link and the password.

Hiding the Secrets from the Source Code

It’s important not to put the text of the username and password inside your code files; it makes it too easy for others to find (including robots), and makes it more difficult to share code with others. Instead, we should put secrets it in a seperate file that we won’t share with others; something that our code file can use.

Here, we’ll try out a standard approach that works in a wide variety of situations: make a .env (“dot env”) file, which has all the environment variables we want to use. Because it has a special extension, this file is easy to tell git to ignore, and many tools know how to work with it automatically.

Make a file called .env and write the username and password into the file as variables like so:

URL=https://uni-bonn.sciebo.de/public.php/webdav  # or whatever the correct address is.
USERNAME=f81JqGZmEHLxMnB                          # the last part of the share link (e.g. from https://uni-bonn.sciebo.de/s/f81JqGZmEHLxMnB)
PASSWORD=mypassword                               # the password

To load the data into python, we can use the python-dotenv to create variables in the operating system from the file (called “environment variables”), and then use the os.environ dictionary to access those variables.

Run the code below to see if you can now access the variables:

Exercises 

load_dotenv(override=True)
env = os.environ
env['URL']  # env['USERNAME] and env['PASSWORD'] should also work, if those variables are found.
'https://uni-bonn.sciebo.de/public.php/webdav'

Exercise: Use the env variables to connect to the shared remote folder, and list all the files in that folder.

Solution 
fs = fsspec.filesystem("webdav", base_url=env['URL'], auth=(env['USERNAME'], env['PASSWORD']))
fs.ls("/", detail=False)
['README.txt']

Exercise: Read the text in the README file. What message is inside the file?

Solution 
fs.read_text("README.txt")
'This is a shared Sciebo folder.\nAnyone with the password can read and edit its contents.\nPlease be responsible.'

Exercise: Write text to a file named “<your_name>.txt” (e.g. “emma.txt”), and put a hello message inside that file, and check that the file was created!

Solution 
fs.write_text("nick.txt", "Hello, Nick.")
12

Exercise: Create a folder named “images” and upload a picture of your favorite animal to the folder. Check that it was uploaded properly.

Solution 
# download and save a photo of a giraffe
r = requests.get("https://media.hornbach.de/hb/packshot/as.47485436.jpg")
img_path = Path("data/giraffe.jpg")
img_path.parent.mkdir(exist_ok=True)
img_path.write_bytes(r.content)

# upload the imge and list the content of the remote folder
fs.makedirs("images", exist_ok=True)
fs.upload("data/giraffe.jpg", "images/giraffe.jpg")
fs.ls("images")
[{'name': 'images/giraffe.jpg',
  'href': '/public.php/webdav/images/giraffe.jpg',
  'size': 80086,
  'created': None,
  'modified': datetime.datetime(2025, 11, 19, 9, 30, 9, tzinfo=datetime.timezone.utc),
  'content_language': None,
  'content_type': 'image/jpeg',
  'etag': '"5930a2454d9c8044d27ce635cc64c0b8"',
  'type': 'file',
  'display_name': None}]

Section 3: Mounting Remote Filesystems as a Network Drive

In Windows, Mac, and Linux, you can mount a remote filesystem so you can browse it in your file explorer. Let’s try it with our shared folder! This doesn’t require any code; when mapping the drive, use the same url, username, and password as we did in the previous exercise.

Exercise: Mount one of the WebDAV folders we connected to as a remote drive, using the “Map Network Drive” option in your file explorer / finder.

Section 4: (Extra) Other Remote Filesystems

The fsspec library supports a wide variety of filesystems, including “SSH” connections (like those used to access linux servers) and “SFTP” connections (like the one provided by the university of Bonn for extra file storage). No matter where the data is, if you have permission to access it, you can use it!