Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • djangoldp-packages/djangoldp
  • decentral1se/djangoldp
  • femmefaytale/djangoldp
  • jvtrudel/djangoldp
4 results
Show changes
Commits on Source (943)
843 additional commits have been omitted to prevent performance issues.
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 2022 additions and 205 deletions
.gitignore
View file @ e97d33ad
dist
build
*.egg-info
*.eggs
*.pyc
*~
*.swp
djangoldp/tests/tests_temp.py
*/.idea/*
.DS_STORE
venv
.gitlab-ci.yml
View file @ e97d33ad
---
image: python:3.6
image: python:3.11
stages:
- test
- release
include:
project: infra/gitlab
ref: master
file: templates/python.ci.yml
test:
stage: test
......@@ -11,21 +12,22 @@ test:
- pip install .[dev]
- python -m unittest djangoldp.tests.runner
except:
- master
- tags
tags:
- sib
- test
publish:
stage: release
before_script:
- git config user.name "${GITLAB_USER_NAME}"
- git config user.email "${GITLAB_USER_EMAIL}"
- git remote set-url origin "https://gitlab-ci-token:${GL_TOKEN}@git.happy-dev.fr/${CI_PROJECT_PATH}.git"
- pip install git+https://github.com/plup/python-semantic-release
- pip install sib-commit-parser
crypto-test:
stage: test
script:
- semantic-release publish
only:
- master
- pip install .[crypto]
- python -m unittest djangoldp_crypto.tests.runner
except:
- tags
tags:
- sib
- test
publish:
cache: []
extends: .publish_pypi
CHANGELOG.md 0 → 100644
View file @ e97d33ad
# Changelog
<!--next-version-placeholder-->
## v3.0.5 (2023-07-24)
### Fix
* Readme ([`8792530`](https://github.com/djangoldp-packages/djangoldp/commit/87925305b2093282230511bceb38978db4b279a2))
## v3.0.4 (2023-07-24)
### Fix
* Readme ([`eeedc33`](https://github.com/djangoldp-packages/djangoldp/commit/eeedc3378ed3f4e454377431da6bb50202efdcdc))
## v3.0.3 (2023-07-24)
### Fix
* Readme ([`73597b6`](https://github.com/djangoldp-packages/djangoldp/commit/73597b65430a4d23306f78def0331bda60857493))
## Unreleased
* Imported CLI along with development template from sib-manager project
Dockerfile 0 → 100644
View file @ e97d33ad
# use with your own settings.yml
FROM python:3.11
LABEL maintainer="Plup <plup@plup.io>"
# get server
RUN pip install djangoldp
# create a server instance
RUN djangoldp initserver ldpserver
WORKDIR /ldpserver
#COPY settings.yml .
RUN djangoldp install
RUN djangoldp configure --with-dummy-admin
# run the server
EXPOSE 8000
CMD ["djangoldp", "runserver"]
LICENSE
View file @ e97d33ad
MIT License
Copyright (c) 2018 Startin blox
Copyright (c) 2018-2023 Startin blox
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
......
MANIFEST.in 0 → 100644
View file @ e97d33ad
graft djangoldp/conf/package_template
graft djangoldp/conf/server_template
graft djangoldp/templates/
README.md
View file @ e97d33ad
## Synopsis
This module is an add-on for Django REST Framework that serves a django model respecting the Linked Data Platform convention.
It aims at enabling people with little development skills to serve their own data, to be used with a LDP application.
# Setup a DjangoLDP server
## Requirements
Check the [official documentation](https://docs.startinblox.com/import_documentation/djangoldp_guide/install-djangoldp-server.html).
* Django (known to work with django 1.11)
* Django Rest Framework
* pyld
* django-guardian
* djangorestframework-guardian
## Installation
1. Install this module and all its dependencies
```
pip install djangoldp
```
2. Create a django project
```
django-admin startproject myldpserver
```
3. Create your django model inside a file myldpserver/myldpserver/models.py
Note that container_path will be use to resolve instance iri and container iri
In the future it could also be used to auto configure django router (e.g. urls.py)
```
from djangoldp.models import Model
class Todo(Model):
name = models.CharField(max_length=255)
deadline = models.DateTimeField()
```
## Settings
3.1. Configure container path (optional)
By default it will be "todos/" with an S for model called Todo
```
<Model>._meta.container_path = "/my-path/"
```
3.2. Configure field visibility (optional)
Note that at this stage you can limit access to certain fields of models using
```
<Model>._meta.serializer_fields (<>list of field names to show>)
```
For example, if you have a model with a related field with type **django.contrib.auth.models.User** you don't want to show personal details or password hashes.
E.g.
```
from django.contrib.auth.models import User
User._meta.serializer_fields = ('username','first_name','last_name')
```
Note that this will be overridden if you explicitly set the fields= parameter as an argument to LDPViewSet.urls(), and filtered if you set the excludes= parameter.
4. Add a url in your urls.py:
```
from django.conf.urls import url
from django.contrib import admin
from djangoldp.views import LDPViewSet
from .models import Todo
urlpatterns = [
url(r'^', include('djangoldp.urls')),
url(r'^admin/', admin.site.urls), # Optional
]
```
* `OIDC_ACCESS_CONTROL_ALLOW_HEADERS`: overrides the access control headers allowed in the `Access-Control-Allow-Headers` CORS header of responses. Defaults to `authorization, Content-Type, if-match, accept, DPoP`
* `ANONYMOUS_USER_NAME` a setting inherited from dependency [Django-Guardian](https://django-guardian.readthedocs.io/en/stable/overview.html)
* `DJANGOLDP_PERMISSIONS`: overrides the list of all permissions on all resources
* `SERIALIZER_CACHE`: toggles the use of a built-in cache in the serialization of containers/resources
* `MAX_RECORDS_SERIALIZER_CACHE`: sets the maximum number of serializer cache records, at which point the cache will be cleared (reset). Defaults to 10,000
* `SEND_BACKLINKS`: enables the searching and sending of [Activities](https://git.startinblox.com/djangoldp-packages/djangoldp/-/wikis/guides/federation) to distant resources linked by users to this server
* `MAX_ACTIVITY_RESCHEDULES`, `DEFAULT_BACKOFF_FACTOR`, `DEFAULT_ACTIVITY_DELAY`, `DEFAULT_REQUEST_TIMEOUT` tweaks the behaviour of the ActivityQueueService
* `STORE_ACTIVITIES`: sets whether to store activities sent and backlinks received, or to treat them as transient (value should be `"VERBOSE"`, `"ERROR"` or `None`). Defaults to `"ERROR"`
* `MAX_RECORDS_ACTIVITY_CACHE`: sets the maximum number of serializer cache records, at which point the cache will be cleared (reset). If set to 0 disables the cache. Defaults to 10,000
* `ENABLE_SWAGGER_DOCUMENTATION`: enables the automatic OpenAPI-based API schema and documentation generation, made available at `http://yourserver/docs/` is the flag is set to True. Default to False
* `DISABLE_LOCAL_OBJECT_FILTER`: disabled the LocalObjectBackendFilter which is processing-time costly and only need activation in federated architecture, so we preferred to add a way to disable it as a workaround for in-progress performances improvements. Default to False
This creates 2 routes for each Model, one for the list, and one with an ID listing the detail of an object.
You could also only use this line in settings.py instead:
```
ROOT_URLCONF = 'djangoldp.urls'
```
5. In the settings.py file, add your application name at the beginning of the application list, and add the following lines
## Synopsis
```
STATIC_ROOT = os.path.join(os.path.dirname(BASE_DIR), 'static')
LDP_RDF_CONTEXT = 'https://cdn.happy-dev.fr/owl/hdcontext.jsonld'
```
This module is an add-on for Django REST Framework that serves a django model respecting the Linked Data Platform convention.
6. You can also register your model for the django administration site
It aims at enabling people with little development skills to serve their own data, to be used with a LDP application.
```
from django.contrib import admin
from .models import Todo
## Check technical documentation
admin.site.register(Todo)
```
* [Using DjangoLDP with your models](./docs/create_model.md)
7. You then need to have your WSGI server pointing on myldpserver/myldpserver/wsgi.py
## Contribute to DjangoLDP
8. You will probably need to create a super user
```
./manage.py createsuperuser
```
### Testing
9. If you have no CSS on the admin screens :
```
./manage.py collectstatic
```
Packaged with DjangoLDP is a tests module, containing unit tests
## Execution
To start the server, `cd` to the root of your Django project and run :
```
python3 manage.py runserver
```
You can extend these tests and add your own test cases by following the examples in the code. You can then run your tests with:
`python -m unittest djangoldp.tests.runner`
## Custom Parameters to LDPViewSet
## Check your datas integrity
### lookup_field
Can be used to use a slug in the url instead of the primary key.
```
LDPViewSet.urls(model=User, lookup_field='username')
```
Because of the way the DjangoLDP's federation work, you can reach some integrity issue within your datas.
### nested_fields
list of ForeignKey, ManyToManyField, OneToOneField and their reverse relations. When a field is listed in this parameter, a container will be created inside each single element of the container.
You can check them with:
In the following example, besides the urls `/members/` and `/members/<pk>/`, two other will be added to serve a container of the skills of the member: `/members/<pk>/skills/` and `/members/<pk>/skills/<pk>/`
```
url(r'^members/', LDPViewSet.urls(model=Member, nested_fields=("skills",))),
```bash
./manage.py check_integrity
```
From the 0.5 we added permissions check by default on every route, so you may encounter 400 errors code on your POST requests. You can disable those checks by specifying the permission_classes as an empty array in our URLs files.
You can ignore some servers:
```
url(r'^posts/', LDPViewSet.urls(model=Post, permission_classes=(), filter_backends = ())),
```bash
./manage.py check_integrity --ignore "https://server/,https://another-server/"
```
## Custom Meta options on models
### Add you own commands to the `check_integrity` from your own package
### rdf_type
### auto_author
This property allows to associate a model with the logged in user.
Create a `check_integrity.py` file within your app folder containing:
```python
class MyModel(models.Model):
author_user = models.ForeignKey(settings.AUTH_USER_MODEL)
class Meta:
auto_author = 'author_user'
```
Now when an instance of `MyModel` is saved, its `author_user` property will be set to the current user.
## permissions
This allows you to add permissions for AnonymousUser, logged in user, author ... in the url:
Currently, there are 3 choices :
* ObjectPermission
* AnonymousReadOnly
* InboxPermissions
Specific permissin classes can be developed to fit special needs.
ObjectPermission give permissions assign in the administration
AnonymousReadOnly gives these permissions:
* Anonymous users: can read all posts
* Logged in users: can read all posts + create new posts
* Author: can read all posts + create new posts + update their own
```
from django.conf.urls import url
from djangoldp.views import LDPViewSet
from djangoldp.permissions import AnonymousReadOnly
urlpatterns = [
url(r'^projects/', ProjectViewSet.urls(permission_classes=(AnonymousReadOnly,))),
url(r'^customers/', LDPViewSet.urls(model=Customer)),
]
```
InboxPermissions is used for, well, notifications:
* Anonymous users: can create notifications but can't read
* Logged in users: can create notifications but can't read
* Inbox owners: can read + update all notifications
```
from django.conf.urls import url
from djangoldp.views import LDPViewSet
from djangoldp.permissions import NotificationsPermissions
urlpatterns = [
url(r'^projects/', ProjectViewSet.urls(permission_classes=(InboxPermissions,))),
url(r'^customers/', LDPViewSet.urls(model=Customer)),
]
```
Important note:
If you need to give permissions to owner's object, don't forget to add auto_author in model's meta
def add_arguments(parser):
parser.add_argument(
"--my-own-argument",
default=False,
nargs="?",
const=True,
help="Some help text",
)
def check_integrity(options):
if(options["my_own_argument"]):
print("You ran a check_integrity with --my-own-argument!")
else:
print("Run me with `./manage.py check_integrity --my-own-argument`")
```
You can see a sample on the `check_integrity.py` file of DjangoLDP.
## License
......
djangoldp/__init__.py
View file @ e97d33ad
from django.db.models import options
__version__ = '0.0.0'
options.DEFAULT_NAMES += ('rdf_type', 'auto_author')
options.DEFAULT_NAMES += (
'lookup_field', 'rdf_type', 'rdf_context', 'auto_author', 'owner_field', 'owner_urlid_field',
'view_set', 'container_path', 'permission_classes', 'serializer_fields', 'serializer_fields_exclude', 'empty_containers',
'nested_fields', 'depth', 'permission_roles', 'inherit_permissions', 'public_field', 'static_version', 'static_params', 'active_field', 'disable_url')
djangoldp/activities/__init__.py 0 → 100644
View file @ e97d33ad
from .objects import *
from .verbs import *
from .services import *
djangoldp/activities/errors.py 0 → 100644
View file @ e97d33ad
class ActivityStreamDecodeError(Exception):
pass
class ActivityStreamTypeError(Exception):
pass
class ActivityStreamValidationError(Exception):
pass
djangoldp/activities/objects.py 0 → 100644
View file @ e97d33ad
import json
import requests
from djangoldp.activities import errors
class Object(object):
attributes = ["type", "id", "name", "to"]
type = "Object"
@classmethod
def from_json(cls, json):
return Object(**json)
def __init__(self, obj=None, **kwargs):
if obj:
self.__init__(**obj.to_activitystream())
for key in self.attributes:
value = kwargs.get(key)
if value is None:
continue
if isinstance(value, dict) and value.get("type"):
value = as_activitystream(value)
self.__setattr__(key, value)
def __str__(self):
content = json.dumps(self, default=encode_activitystream)
return "<{type}: {content}>".format(type=self.type, content=content)
def to_json(self, context=False):
'''
converts Object to json
context parameter indicates whether @context should be added to the object (for JSON-LD parsing)
'''
values = {}
for attribute in self.attributes:
value = getattr(self, attribute, None)
if value is None:
continue
if isinstance(value, Object):
value = value.to_json()
# if getattr(value, "__iter__", None):
# value = [item.to_json() for item in value]
values[attribute] = value
to = values.get("to")
if isinstance(to, str):
values["to"] = [to]
elif getattr(to, "__iter__", None):
values["to"] = []
for item in to:
if isinstance(item, str):
values["to"].append(item)
if isinstance(item, Object):
values["to"].append(item.id)
if context:
values["@context"] = "https://www.w3.org/ns/activitystreams"
return values
def to_activitystream(self):
return self
class Actor(Object):
attributes = Object.attributes + [
"target",
"publicKey",
"preferredUsername",
"following",
"followers",
"outbox",
"inbox",
]
type = "Actor"
def send(self, activity):
res = requests.post(self.inbox, json=activity.to_json(context=True))
if res.status_code != 200:
raise Exception
class Person(Actor):
type = "Person"
class Service(Actor):
type = "Service"
class Group(Actor):
type = "Group"
class Collection(Object):
attributes = Object.attributes + ["items", "totalItems"]
type = "Collection"
def __init__(self, iterable=None, **kwargs):
self._items = []
Object.__init__(self, **kwargs)
if iterable is None:
return
self.items = iterable
@property
def items(self):
return self._items
@items.setter
def items(self, iterable):
for item in iterable:
if isinstance(item, Object):
self._items.append(item)
elif isinstance(item, str):
self._items.append(item)
elif getattr(item, "to_activitystream", None):
item = as_activitystream(item.to_activitystream())
self._items.append(item)
else:
raise Exception("invalid ActivityStream object: {item}".format(item=item))
def to_json(self, **kwargs):
json = Object.to_json(self, **kwargs)
items = [item.to_json() if isinstance(item, Object) else item
for item in self.items]
json.update({
"items": items
})
return json
class OrderedCollection(Collection):
attributes = Object.attributes + ["orderedItems", "totalItems"]
type = "OrderedCollection"
@property
def totalItems(self):
return len(self.items)
@totalItems.setter
def totalItems(self, value):
pass
@property
def orderedItems(self):
return self.items
@orderedItems.setter
def orderedItems(self, iterable):
self.items = iterable
def to_json(self, **kwargs):
json = Collection.to_json(self, **kwargs)
json["orderedItems"] = json["items"]
del json["items"]
return json
class Link(Object):
type = 'Link'
attributes = ['link']
def to_json(self, **kwargs):
return self.link
#########
# Utils #
#########
ALLOWED_TYPES = {
"Object": Object,
"Actor": Actor,
"Service": Service,
"Person": Person,
"Link": Link,
"Collection": Collection,
"OrderedCollection": OrderedCollection
}
def as_activitystream(obj):
type = obj.get("type")
if type:
if type in ALLOWED_TYPES:
obj = ALLOWED_TYPES[type](**obj)
return obj
raise errors.ActivityStreamDecodeError("Invalid Type {0}".format(type))
return obj
def encode_activitystream(obj):
if isinstance(obj, Object):
return obj.to_json()
raise errors.ActivityStreamTypeError("Unknown ActivityStream Type")
djangoldp/activities/services.py 0 → 100644
View file @ e97d33ad
This diff is collapsed.
djangoldp/activities/verbs.py 0 → 100644
View file @ e97d33ad
from copy import copy
from djangoldp.activities import errors
from djangoldp.activities.objects import ALLOWED_TYPES, Object, Actor
class Activity(Object):
attributes = Object.attributes + ["actor", "object"]
type = "Activity"
# dictionary defining required attributes -> tuple of acceptable types
required_attributes = {
"actor": (Actor, str),
"object": dict
}
def get_audience(self):
audience = []
for attr in ["to", "bto", "cc", "bcc", "audience"]:
value = getattr(self, attr, None)
if not value:
continue
if isinstance(value, str):
value = [value]
audience += value
return set(audience)
def strip_audience(self):
new = copy(self)
if getattr(new, "bto", None):
delattr(new, "bto")
if getattr(new, "bcc", None):
delattr(new, "bcc")
return new
def _validate_type_id_defined(self, value):
'''recursively ensures that all nested dict items define @id and @type attributes'''
for item in value.items():
if isinstance(item[1], dict):
item_value = item[1]
if '@type' not in item_value or '@id' not in item_value:
raise errors.ActivityStreamValidationError("all sub-objects passed in activity object must define @id and @type tags")
self._validate_type_id_defined(item_value)
def validate(self):
for attr in self.required_attributes.keys():
if not isinstance(getattr(self, attr, None), self.required_attributes[attr]):
raise errors.ActivityStreamValidationError("required attribute " + attr + " of type "
+ str(self.required_attributes[attr]))
# validate that every dictionary stored in object has @id and @type
self._validate_type_id_defined(self.__getattribute__("object"))
class Add(Activity):
type = "Add"
attributes = Activity.attributes + ["target"]
required_attributes = {**Activity.required_attributes, "target": dict}
class Remove(Activity):
type = "Remove"
attributes = Activity.attributes + ["target", "origin"]
def validate(self):
super().validate()
if not getattr(self, "target", None) and not getattr(self, "origin", None):
raise errors.ActivityStreamValidationError("Invalid activity, no target or origin given")
if getattr(self, "target", None) is not None:
if not isinstance(self.target, dict):
raise errors.ActivityStreamValidationError("Invalid target type, must be a dict")
if getattr(self, "origin", None) is not None:
if not isinstance(self.origin, dict):
raise errors.ActivityStreamValidationError("Invalid origin type, must be a dict")
class Create(Activity):
type = "Create"
class Update(Activity):
type = "Update"
class Delete(Activity):
type = "Delete"
attributes = Activity.attributes + ["origin"]
class Follow(Activity):
type = "Follow"
def validate(self):
super().validate()
if isinstance(self.actor, Actor) and (self.actor.inbox is None and self.actor.id is None):
raise errors.ActivityStreamValidationError("Must pass inbox or id with the actor to follow")
ALLOWED_TYPES.update({
"Activity": Activity,
"Add": Add,
"Remove": Remove,
"Create": Create,
"Update": Update,
"Delete": Delete,
"Follow": Follow
})
djangoldp/admin.py
View file @ e97d33ad
from csv import DictWriter
from django.contrib import admin
from .models import LDPSource, LDNotification
from django.contrib.auth.admin import UserAdmin
from django.core.exceptions import FieldDoesNotExist
from django.http import HttpResponse
from guardian.admin import GuardedModelAdmin
from djangoldp.models import Activity, ScheduledActivity, Follower
from djangoldp.activities.services import ActivityQueueService
class DjangoLDPAdmin(GuardedModelAdmin):
'''
An admin model representing a federated object. Inherits from GuardedModelAdmin to provide Django-Guardian
object-level permissions
'''
actions = ['export_csv']
export_fields = []
def resolve_verbose_name(self, field_path):
field = self
for field_name in field_path.split('__'):
try:
field = field.model._meta.get_field(field_name)
except FieldDoesNotExist:
return None
return field.verbose_name
@admin.action(description="Export CSV")
def export_csv(self, request, queryset):
response = HttpResponse(content_type="text/csv")
response['Content-Disposition'] = f'attachment; filename="{self.model.__name__}.csv"'
# only keep fields that can be resolved, keep only urlid if none
field_list = list(filter(self.resolve_verbose_name, self.export_fields or self.list_display)) or ['urlid']
headers = {field:self.resolve_verbose_name(field) for field in field_list}
writer = DictWriter(response, fieldnames=field_list)
writer.writerow(headers)
writer.writerows(queryset.values(*field_list))
return response
class DjangoLDPUserAdmin(UserAdmin, DjangoLDPAdmin):
'''An extension of UserAdmin providing the functionality of DjangoLDPAdmin'''
list_display = ('urlid', 'email', 'first_name', 'last_name', 'date_joined', 'last_login', 'is_staff')
search_fields = ['urlid', 'email', 'first_name', 'last_name']
ordering = ['urlid']
def get_fieldsets(self, request, obj=None):
fieldsets = super().get_fieldsets(request, obj)
federated_fields = ['urlid', 'allow_create_backlink']
if self.exclude is not None:
federated_fields = list(set(federated_fields) - set(self.exclude))
for fieldset in fieldsets:
federated_fields = list(set(federated_fields) - set(fieldset[1]['fields']))
if len(federated_fields) == 0:
return fieldsets
fieldsets = [('Federation', {'fields': federated_fields})] + list(fieldsets)
return fieldsets
@admin.action(description='Resend activity')
def resend_activity(modeladmin, request, queryset):
for a in queryset:
ActivityQueueService.send_activity(a.external_id, a.to_activitystream())
resend_activity.short_description = 'Resend activity'
@admin.register(Activity, ScheduledActivity)
class ActivityAdmin(DjangoLDPAdmin):
fields = ['urlid', 'type', 'local_id', 'external_id', 'created_at', 'success', 'payload_view', 'response_code',
'response_location', 'response_body_view']
list_display = ['created_at', 'type', 'local_id', 'external_id', 'success', 'response_code']
readonly_fields = ['created_at', 'payload_view', 'response_location', 'response_code', 'response_body_view']
search_fields = ['urlid', 'type', 'local_id', 'external_id', 'response_code']
actions = [resend_activity]
def payload_view(self, obj):
return str(obj.to_activitystream())
def response_body_view(self, obj):
return str(obj.response_to_json())
@admin.register(Follower)
class FollowerAdmin(DjangoLDPAdmin):
fields = ['urlid', 'object', 'inbox', 'follower']
list_display = ['urlid', 'object', 'inbox', 'follower']
search_fields = ['object', 'inbox', 'follower']
admin.site.register(LDPSource)
admin.site.register(LDNotification)
djangoldp/apps.py
View file @ e97d33ad
import os
from django.apps import AppConfig
class DjangoldpConfig(AppConfig):
name = 'djangoldp'
def ready(self):
self.auto_register_model_admin()
self.start_activity_queue()
# Patch guardian core to avoid prefetching permissions several times
from guardian.core import ObjectPermissionChecker
ObjectPermissionChecker._prefetch_cache_orig = ObjectPermissionChecker._prefetch_cache
def _prefetch_cache(self):
if hasattr(self.user, "_guardian_perms_cache"):
self._obj_perms_cache = self.user._guardian_perms_cache
return
self._prefetch_cache_orig()
ObjectPermissionChecker._prefetch_cache = _prefetch_cache
def start_activity_queue(self):
from djangoldp.activities.services import ActivityQueueService
if os.environ.get('RUN_MAIN') is not None:
ActivityQueueService.start()
def auto_register_model_admin(self):
'''
Automatically registers Model subclasses in the admin panel (which have not already been added manually)
'''
from importlib import import_module
from django.conf import settings
from django.contrib import admin
from djangoldp.admin import DjangoLDPAdmin
from djangoldp.urls import get_all_non_abstract_subclasses
from djangoldp.models import Model
for package in settings.DJANGOLDP_PACKAGES:
try:
import_module('{}.admin'.format(package))
except ModuleNotFoundError:
pass
for package in settings.DJANGOLDP_PACKAGES:
try:
import_module('{}.models'.format(package))
except ModuleNotFoundError:
pass
for model in get_all_non_abstract_subclasses(Model):
if not admin.site.is_registered(model):
admin.site.register(model, DjangoLDPAdmin)
djangoldp/check_integrity.py 0 → 100644
View file @ e97d33ad
'''
DjangoLDP Check Integrity
Usage `./manage.py check_integrity --help`
'''
from django.apps import apps
from django.conf import settings
from djangoldp.models import LDPSource
from urllib.parse import urlparse
import requests
# Helper command for argument type checking
def is_string(target):
if isinstance(target, str):
return target
return False
# Helper command to check status code of a target
def is_alive(target, status_code = 200):
return requests.get(target).status_code == status_code
# Add argument to the `check_integrity` command
def add_arguments(parser):
parser.add_argument(
"--ignore",
action="store",
default=False,
type=is_string,
help="Ignore any server, comma separated",
)
parser.add_argument(
"--ignore-faulted",
default=False,
nargs="?",
const=True,
help="Ignore eventual faulted",
)
parser.add_argument(
"--ignore-404",
default=False,
nargs="?",
const=True,
help="Ignore eventual 404",
)
parser.add_argument(
"--fix-faulted-resources",
default=False,
nargs="?",
const=True,
help="Fix faulted resources",
)
parser.add_argument(
"--fix-404-resources",
default=False,
nargs="?",
const=True,
help="Fix 404 resources",
)
parser.add_argument(
"--fix-offline-servers",
default=False,
nargs="?",
const=True,
help="Remove resources from offline servers",
)
# Define our own checks
def check_integrity(options):
models = apps.get_models()
ignored = set()
if(options["ignore"]):
for target in options["ignore"].split(","):
ignored.add(urlparse(target).netloc)
if(len(ignored) > 0):
print("Ignoring servers:")
for server in ignored:
print("- "+server)
resources = set()
resources_map = dict()
base_urls = set()
for model in models:
for obj in model.objects.all():
if hasattr(obj, "urlid"):
if(obj.urlid):
if(not obj.urlid.startswith(settings.BASE_URL)):
url = urlparse(obj.urlid).netloc
if(url not in ignored):
resources.add(obj.urlid)
resources_map[obj.urlid] = obj
base_urls.add(url)
if(len(base_urls) > 0):
print("Found "+str(len(resources_map))+" distant resources on "+str(len(models))+" models")
print("Servers that I have backlinks to:")
for server in base_urls:
print("- "+server)
else:
print("I don't have any backlink")
source_urls = set()
for source in LDPSource.objects.all():
source_urls.add(urlparse(source.urlid).netloc)
if(len(source_urls) > 0):
print("Servers that I'm allowed to get federated to:")
for server in source_urls:
print("- "+server)
else:
print("I'm not federated")
difference_servers = base_urls.difference(source_urls)
if(len(difference_servers) > 0):
print("Servers that I should not get aware of:")
for server in difference_servers:
print("- "+server)
# Handle faulted resources
if(not options["ignore_faulted"]):
faulted_resources = set()
for server in difference_servers:
for resource in resources:
if(urlparse(resource).netloc in server):
faulted_resources.add(resource)
if(len(faulted_resources) > 0):
print("Resources in fault:")
for resource in faulted_resources:
print("- "+resource)
else:
print("No resource are in fault")
if(options["fix_faulted_resources"]):
for resource in faulted_resources:
try:
resources_map[resource].delete()
except:
pass
print("Fixed faulted resources")
else:
print("Fix them with `./manage.py check_integrity --fix-faulted-resources`")
else:
print("I accept datas for every of those servers")
# Handle 404 resources
if(not options["ignore_404"]):
resources_404 = set()
resources_servers_offline = set()
for resource in resources:
try:
if(is_alive(resource, 404)):
resources_404.add(resource)
except:
resources_servers_offline.add(resource)
if(len(resources_404) > 0):
print("Faulted resources, 404:")
for resource in resources_404:
print("- "+resource)
if(options["fix_404_resources"]):
for resource in resources_404:
try:
resources_map[resource].delete()
except:
pass
print("Fixed 404 resources")
else:
print("Fix them with `./manage.py check_integrity --fix-404-resources`")
if(len(resources_servers_offline) > 0):
print("Faulted resources, servers offline:")
for resource in resources_servers_offline:
print("- "+resource)
if(options["fix_offline_servers"]):
for resource in resources_servers_offline:
try:
resources_map[resource].delete()
except:
pass
print("Fixed resources on offline servers")
else:
print("Fix them with `./manage.py check_integrity --fix-offline-servers`")
else:
print("No 404 in known resources")
\ No newline at end of file
djangoldp/cli.py 0 → 100644
View file @ e97d33ad
import click
import sys
import yaml
import subprocess
from pkg_resources import resource_filename
from pathlib import Path
from django.core import management
from django.core.management.base import CommandError
from . import __version__
# click entrypoint
@click.group()
@click.version_option(__version__)
def main():
"""DjangoLDP CLI"""
@main.command()
@click.argument('name', nargs=1, required=False)
def initserver(name):
"""Start a DjangoLDP server."""
try:
# use directly pwd
directory = Path.cwd()
if name:
# create a directory from project name in pwd
directory = Path.cwd() / name
directory.mkdir(parents=False, exist_ok=False)
# get the template path
template = resource_filename(__name__, 'conf/server_template')
# wrap the default django-admin startproject command
# this call import django settings and configure it
# see: https://docs.djangoproject.com/fr/2.2/topics/settings/#calling-django-setup-is-required-for-standalone-django-usage
# see: https://github.com/django/django/blob/stable/2.2.x/django/core/management/templates.py#L108
# fix: in 2.2 gabarit files options has been renamed: https://github.com/django/django/blob/stable/2.2.x/django/core/management/templates.py#L53
management.call_command('startproject', name, directory, template=template, files=['settings.yml'])
except FileExistsError:
click.echo(f'Error: the folder {directory} already exists')
sys.exit(1)
except CommandError as e:
click.echo(f'Error: {e}')
directory.rmdir()
sys.exit(1)
@main.command()
@click.argument('name', nargs=1)
def startpackage(name):
"""Start a DjangoLDP package."""
try:
# set directory
directory = Path.cwd() / name
# get the template path
template = resource_filename(__name__, 'conf/package_template')
# create dir
directory.mkdir(parents=False, exist_ok=False)
# wrap the default startapp command
management.call_command('startapp', name, directory, template=template)
except FileExistsError:
click.echo(f'Error: the folder {directory} already exists')
sys.exit(1)
except CommandError as e:
click.echo(f'Error: {e}')
sys.exit(1)
@main.command()
def install():
"""Install project dependencies."""
try:
# load dependencies from config file
path = Path.cwd() / 'settings.yml'
with open(path, 'r') as f:
dependencies = yaml.safe_load(f).get('dependencies', [])
# install them by calling pip command
cmd = [sys.executable, "-m", "pip", "install", "--upgrade"]
try:
cmd.extend(dependencies)
subprocess.run(cmd).check_returncode()
click.echo('Installation done!')
except TypeError:
click.echo('No dependency to install')
except FileNotFoundError:
click.echo('Config error: no settings.yml file in this directory')
sys.exit(1)
except subprocess.CalledProcessError as e:
click.echo(f'Installation error: {e}')
sys.exit(1)
@main.command()
@click.option('--with-admin', 'admin', help='Create an administrator user with email.')
@click.option('--email', help='Provide an email for administrator.')
@click.option('--with-dummy-admin', 'dummy_admin', is_flag=True, help='Create a default "admin" user.')
def configure(admin, dummy_admin, email):
"""Configure the project."""
try:
# shortcut to the djangoldp.management command
path = str(Path.cwd() / 'manage.py')
cmd = [sys.executable, path, 'configure']
if admin:
if not email:
click.echo('Error: missing email for admin user')
return
cmd.extend(['--with-admin', admin, '--email', email])
elif dummy_admin:
cmd.append('--with-dummy-admin')
subprocess.run(cmd).check_returncode()
click.echo('Configuration done!')
except subprocess.CalledProcessError as e:
click.echo(f'Configuration error: {e}')
sys.exit(1)
@main.command()
def runserver():
"""Run the Django embeded webserver."""
try:
# shortcut to the djangoldp.management command
path = str(Path.cwd() / 'manage.py')
cmd = [sys.executable, path, 'runserver', '0.0.0.0:8000']
subprocess.run(cmd).check_returncode()
except subprocess.CalledProcessError as e:
click.echo(f'Execution error: {e}')
sys.exit(1)
djangoldp/conf/__init__.py 0 → 100644
View file @ e97d33ad
djangoldp/conf/default_settings.py 0 → 100644
View file @ e97d33ad
"""This module override some of default django configuration from the global settings."""
from django.conf.global_settings import *
####################
# LDP #
####################
LDP_RDF_CONTEXT = 'https://cdn.startinblox.com/owl/context.jsonld'
MAX_ACTIVITY_RESCHEDULES = 3
DEFAULT_BACKOFF_FACTOR = 1
DEFAULT_ACTIVITY_DELAY = 0.2
DEFAULT_REQUEST_TIMEOUT = 10
LDP_INCLUDE_INNER_PERMS = False
####################
# CORE #
####################
# https://en.wikipedia.org/wiki/List_of_tz_zones_by_name (although not all
# systems may support all possibilities). When USE_TZ is True, this is
# interpreted as the default user time zone.
TIME_ZONE = 'UTC'
# If you set this to True, Django will use timezone-aware datetimes.
USE_TZ = True
# Language code for this installation. All choices can be found here:
# http://www.i18nguy.com/unicode/language-identifiers.html
LANGUAGE_CODE = 'en-us'
# Database connection info. If left empty, will default to the dummy backend.
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': 'db.sqlite3',
}
}
# List of strings representing installed apps.
INSTALLED_APPS = [
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'djangoldp',
'guardian'
]
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
# Default server url
SITE_URL = 'http://localhost:8000'
BASE_URL = SITE_URL
# Default URL conf
ROOT_URLCONF = 'djangoldp.urls'
# Absolute filesystem path to the directory that will hold user-uploaded files.
# Example: "/var/www/example.com/media/"
MEDIA_ROOT = ''
# URL that handles the media served from MEDIA_ROOT.
# Examples: "http://example.com/media/", "http://media.example.com/"
MEDIA_URL = '/media/'
# Absolute path to the directory static files should be collected to.
# Example: "/var/www/example.com/static/"
STATIC_ROOT = None
# URL that handles the static files served from STATIC_ROOT.
# Example: "http://example.com/static/", "http://static.example.com/"
STATIC_URL = '/static/'
# ETAGS config
USE_ETAGS = True
# Default X-Frame-Options header value
X_FRAME_OPTIONS = 'DENY'
# The Python dotted path to the WSGI application that Django's internal server
# (runserver) will use. If `None`, the return value of
# 'django.core.wsgi.get_wsgi_application' is used, thus preserving the same
# behavior as previous versions of Django. Otherwise this should point to an
# actual WSGI application object.
WSGI_APPLICATION = 'server.wsgi.application'
##############
# MIDDLEWARE #
##############
# List of middleware to use. Order is important; in the request phase, these
# middleware will be applied in the order given, and in the response
# phase the middleware will be applied in reverse order.
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
'djangoldp.middleware.AllowRequestedCORSMiddleware',
'django.middleware.gzip.GZipMiddleware',
'django_brotli.middleware.BrotliMiddleware'
]
##################
# REST FRAMEWORK #
##################
REST_FRAMEWORK = {
'DEFAULT_SCHEMA_CLASS': 'drf_spectacular.openapi.AutoSchema',
'DEFAULT_PAGINATION_CLASS': 'djangoldp.pagination.LDPPagination'
}
###################
# DRF SPECTACULAR #
###################
ENABLE_SWAGGER_DOCUMENTATION = False
SPECTACULAR_SETTINGS = {
'TITLE': 'DjangoLDP Based API Description',
'DESCRIPTION': 'Here you will find the list of all endpoints available on your Djangoldp-based server instance and\
the available methods, needed parameters and requests examples. The list of available endpoints depend on your instance configuration\
especially the list of bloxes and associated django models activated.',
'VERSION': '2.1.37',
'SERVE_INCLUDE_SCHEMA': False
}
############
# SESSIONS #
############
# Same site policy
DCS_SESSION_COOKIE_SAMESITE = 'none'
##################
# AUTHENTICATION #
##################
AUTHENTICATION_BACKENDS = ['django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend']
OIDC_ACCESS_CONTROL_ALLOW_HEADERS = 'Content-Type, if-match, accept, authorization, DPoP, cache-control, pragma, prefer'
# The minimum number of seconds a password reset link is valid for
PASSWORD_RESET_TIMEOUT = 60 * 60 * 24 * 3
DISABLE_LOCAL_OBJECT_FILTER = False
GUARDIAN_AUTO_PREFETCH = True
DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
djangoldp/conf/ldpsettings.py 0 → 100644
View file @ e97d33ad
import os
import sys
import yaml
import logging
from django.core.exceptions import ImproperlyConfigured
from django.conf import settings as django_settings
from pathlib import Path
from collections import OrderedDict
from typing import Iterable
from importlib import import_module
from . import default_settings
logger = logging.getLogger(__name__)
def configure(filename='settings.yml'):
"""Helper function to configure django from LDPSettings."""
yaml_config = None
try:
with open(filename, 'r') as f:
yaml_config = yaml.safe_load(f)
except FileNotFoundError:
logger.info('Starting project without configuration file')
# ref: https://docs.djangoproject.com/fr/2.2/topics/settings/#custom-default-settings
ldpsettings = LDPSettings(yaml_config)
django_settings.configure(ldpsettings)
class LDPSettings(object):
"""Class managing the DjangoLDP configuration."""
def __init__(self, config):
"""Build a Django Setting object from a dict."""
if django_settings.configured:
raise ImproperlyConfigured('Settings have been configured already')
self._config = config
self._explicit_settings = set(default_settings.__dict__.keys()) #default settings are explicit
self._settings = self.build_settings()
def build_settings(self, extend=['INSTALLED_APPS', 'MIDDLEWARE']):
"""
Look for the parameters in multiple places.
Each step overrides the value of the previous key found. Except for "extend" list. Those value must be lists and all values found are added to these lists without managing duplications.
Resolution order of the configuration:
1. Core default settings
2. Packages settings
3. Code from a local settings.py file
4. YAML config file
"""
# helper loop
def update_with(config):
for setting, value in config.items():
self._explicit_settings.add(setting)
if setting in extend:
settings[setting].extend(value)
elif not setting.startswith('_'):
settings.update({setting: value})
# start from default core settings
settings = default_settings.__dict__.copy()
logger.debug(f'Building settings from core defaults')
# INSTALLED_APPS starts empty
settings['INSTALLED_APPS'] = []
# look settings from packages in the order they are given (local overrides installed)
for pkg in self.DJANGOLDP_PACKAGES:
# FIXME: There is something better to do here with the sys.modules path
try:
# override with values from installed package
mod = import_module(f'{pkg}.djangoldp_settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from installed package {pkg}')
except ModuleNotFoundError:
pass
try:
# override with values from local package
mod = import_module(f'{pkg}.{pkg}.djangoldp_settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from local package {pkg}')
except ModuleNotFoundError:
pass
# look in settings.py file in directory
try:
mod = import_module('settings')
update_with(mod.__dict__)
logger.debug(f'Updating settings from local settings.py file')
except ModuleNotFoundError:
pass
# look in YAML config file 'server' section
try:
conf = self._config.get('server', {})
update_with(conf)
logger.debug(f'Updating settings with project config')
except KeyError:
pass
# In the end adds the INSTALLED_APPS from the core
settings['INSTALLED_APPS'].extend(getattr(default_settings,'INSTALLED_APPS'))
return settings
@property
def DJANGOLDP_PACKAGES(self):
"""Returns the list of LDP packages configured."""
pkg = self._config.get('ldppackages', [])
return [] if pkg is None else pkg
@property
def INSTALLED_APPS(self):
"""Return the installed apps and the LDP packages."""
# get ldp packages (they are django apps)
apps = self.DJANGOLDP_PACKAGES.copy()
# add the default apps
apps.extend(self._settings['INSTALLED_APPS'])
# As settings come from different origins duplicuation is likeliy to happen
return list(OrderedDict.fromkeys(apps))
def __getattr__(self, param):
"""Return the requested parameter from cached settings."""
if param.startswith('_') or param.islower():
# raise the django exception for inexistent parameter
raise AttributeError(f'"{param}" is not compliant to django settings format')
try:
return self._settings[param]
except KeyError:
# raise the django exception for inexistent parameter
raise AttributeError(f'no "{param}" parameter found in settings')
def is_overridden(self, setting):
return setting in self._explicit_settings
\ No newline at end of file